pyhson大数据处理_[上交大] moxing API拷贝大数据集报错

`jobid :

MA-ESRGAN-Ascend-12-21-21-14 | jobc3990c8f`

log : https://jbox.sjtu.edu.cn/l/z5id4s

使用moxing API拷贝较大数据集时报错(数据集图片大概280 k张图片,需要在`tf.data.DataSet`中进一步做数据增强)

```

multiprocessing.pool.RemoteTraceback:

"""

Traceback (most recent call last):

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/util/multiprocessing.py", line 60, in run

super(Process, self).run()

File "/usr/local/ma/python3.7/lib/python3.7/multiprocessing/process.py", line 99, in run

self._target(*self._args, **self._kwargs)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 2053, in _copy_by_queue

atom=atom, skip_not_found=skip_not_found)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/util/runtime.py", line 254, in wrapper

return func(*args, **kwargs)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 246, in wrapper

return func(*args, **kwargs)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 1671, in copy

_download_obs(obs_client, src_bucket_name, src_object_key, dst_object_key)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 1701, in _download_obs

_download_obs_by_stream(obs_client, bucket_name, object_key, local_file)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 1730, in _download_obs_by_stream

with open(local_file, 'wb') as f:

OSError: [Errno 28] No space left on device: 'tmp/data/npz/DIV2K/DIV2K/HR_sub/0507_s011.png'

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/home/work/user-job-dir/00-access/train.py", line 217, in

main()

File "/home/work/user-job-dir/00-access/train.py", line 89, in main

mox.file.copy_parallel(FLAGS.data_url, FLAGS.native_data)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 2021, in copy_parallel

p.join()

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/util/multiprocessing.py", line 75, in join

self._error_callback.check()

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/util/multiprocessing.py", line 49, in check

raise e

OSError: [Errno 28] No space left on device: 'tmp/data/npz/DIV2K/DIV2K/HR_sub/0507_s011.png'

multiprocessing.pool.RemoteTraceback:

"""

Traceback (most recent call last):

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/util/multiprocessing.py", line 60, in run

super(Process, self).run()

File "/usr/local/ma/python3.7/lib/python3.7/multiprocessing/process.py", line 99, in run

self._target(*self._args, **self._kwargs)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 2053, in _copy_by_queue

atom=atom, skip_not_found=skip_not_found)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/util/runtime.py", line 254, in wrapper

return func(*args, **kwargs)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 246, in wrapper

return func(*args, **kwargs)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 1671, in copy

_download_obs(obs_client, src_bucket_name, src_object_key, dst_object_key)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 1701, in _download_obs

_download_obs_by_stream(obs_client, bucket_name, object_key, local_file)

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/file/file_io.py", line 1730, in _download_obs_by_stream

with open(local_file, 'wb') as f:

OSError: [Errno 28] No space left on device: 'tmp/data/npz/DIV2K/DIV2K/HR_sub/0507_s025.png'

"""

The above exception was the direct cause of the following exception:

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/local/ma/python3.7/lib/python3.7/multiprocessing/util.py", line 334, in _exit_function

p.join()

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/util/multiprocessing.py", line 75, in join

self._error_callback.check()

File "/usr/local/ma/python3.7/lib/python3.7/site-packages/moxing/framework/util/multiprocessing.py", line 49, in check

raise e

OSError: [Errno 28] No space left on device: 'tmp/data/npz/DIV2K/DIV2K/HR_sub/0507_s025.png'

[Modelarts Service Log]2020-12-21 13:24:59,743 - ERROR - FMK of device3 (pid: [171]) has exited with non-zero code: 1

[Modelarts Service Log]2020-12-21 13:24:59,744 - INFO - Begin destroy FMK processes

[Modelarts Service Log]2020-12-21 13:24:59,744 - INFO - FMK of device3 (pid: [171]) has exited

[Modelarts Service Log]2020-12-21 13:24:59,744 - INFO - End destroy FMK processes

=== begin proc exit ===

=== begin stop slogd ===

=== end pro exit ===

[ModelArts Service Log]modelarts-pipe: total length: 19897

[Modelarts Service Log]Training end with return code: 1

[Modelarts Service Log]training end at 2020-12-21-13:25:00

[Modelarts Service Log]Training completed.

```

本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！