Question

是否有人成功将大型数据存储类型备份到云存储？这是一个实验性功能，因此谷歌端的支持非常粗略。

我们想要备份到云存储的类型（最终目标是从云存储中提取到大查询）目前的大小为1.2 TB。

- description: BackUp
  url: /_ah/datastore_admin/backup.create?name=OurApp&filesystem=gs&gs_bucket_name=OurBucket&queue=backup&kind=LargeKind
  schedule: every day 00:00
  timezone: America/Regina
  target: ah-builtin-python-bundle

我们一直遇到以下错误消息：

Traceback (most recent call last):
  File "/base/data/home/apps/s~steprep-prod-hrd/prod-339.366560204640641232/lib/mapreduce/handlers.py", line 182, in handle
    input_reader, shard_state, tstate, quota_consumer, ctx)
  File "/base/data/home/apps/s~steprep-prod-hrd/prod-339.366560204640641232/lib/mapreduce/handlers.py", line 263, in process_inputs
    entity, input_reader, ctx, transient_shard_state):
  File "/base/data/home/apps/s~steprep-prod-hrd/prod-339.366560204640641232/lib/mapreduce/handlers.py", line 318, in process_data
    output_writer.write(output, ctx)
  File "/base/data/home/apps/s~steprep-prod-hrd/prod-339.366560204640641232/lib/mapreduce/output_writers.py", line 711, in write
    ctx.get_pool("file_pool").append(self._filename, str(data))
  File "/base/data/home/apps/s~steprep-prod-hrd/prod-339.366560204640641232/lib/mapreduce/output_writers.py", line 266, in append
    self.flush()
  File "/base/data/home/apps/s~steprep-prod-hrd/prod-339.366560204640641232/lib/mapreduce/output_writers.py", line 288, in flush
    f.write(data)
  File "/python27_runtime/python27_lib/versions/1/google/appengine/api/files/file.py", line 297, in __exit__
    self.close()
  File "/python27_runtime/python27_lib/versions/1/google/appengine/api/files/file.py", line 291, in close
    self._make_rpc_call_with_retry('Close', request, response)
  File "/python27_runtime/python27_lib/versions/1/google/appengine/api/files/file.py", line 427, in _make_rpc_call_with_retry
    _make_call(method, request, response)
  File "/python27_runtime/python27_lib/versions/1/google/appengine/api/files/file.py", line 250, in _make_call
    rpc.check_success()
  File "/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 570, in check_success
    self.__rpc.CheckSuccess()
  File "/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
    raise self.exception
DeadlineExceededError: The API call file.Close() took too long to respond and was cancelled.

Answer 1

对于从gae到云存储的写入操作，似乎有未记录的时间限制为30秒。这也适用于在后端进行的写操作，因此您可以从gae创建最大文件大小云存储取决于您的吞吐量。我们的解决方案是拆分文件;每次作家任务接近20秒，它关闭当前文件并打开一个新文件，然后我们在本地加入这些文件。对我们来说，这会产生大约500KB（压缩）的文件，因此这对您来说可能不是一个可接受的解决方案......

备份大型数据存储类型（1TB +）到谷歌云存储

1 个答案: