Question

我在完成写入谷歌存储的mapreduce作业时遇到了一个奇怪的错误，有没有人见过这个？

工作的最终结果'158354152558 ......'是'成功'

...

File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduc/handlers.py", line 539, in _finalize_job
mapreduce_spec.mapper.output_writer_class().finalize_job(mapreduce_state)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/output_writers.py", line 571, in finalize_job
files.finalize(create_filename)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 568, in finalize
f.close(finalize=True)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 291, in close
self._make_rpc_call_with_retry('Close', request, response)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 427, in _make_rpc_call_with_retry
_make_call(method, request, response)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 252, in _make_call
_raise_app_error(e)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 186, in _raise_app_error
raise UnknownError()
UnknownError

Answer 1

使用它之后，我发现云存储上的打开文件必须在不到1小时内完成，否则会因为这个可爱的UnknownError而失败。

我缓解了增加分片数量以使映射更快的问题，并将output_sharding策略更改为“input”，每个分片创建一个文件。

UnknownError最终确定mapreduce作业

1 个答案: