Question

我正在尝试使用entities和datastore向Python插入150k +行/ datastore emulator。

    def add_rows(self, val_dicts):
        # val_dicts is a list of dictionaries for updating the entities

        with self.client.transaction():

            entities = [Entity(self.client.key(self.kind)) for i in range(len(val_dicts))]

            for entity, update_dict in zip(entities, val_dicts):
                entity.update(update_dict)

            self.client.put_multi(entities)

问题是put_multi()需要花费大量时间来插入entities。我想知道如何在datastore插入entities来提高效率。

更新。等待很长时间后put_multi() 完成后，发生了错误，

self.client.put_multi(entities)
../../.local/lib/python3.5/site-packages/google/cloud/datastore/batch.py:293: in __exit__
self.commit()
../../.local/lib/python3.5/site-packages/google/cloud/datastore/transaction.py:220: in commit
super(Transaction, self).commit()
../../.local/lib/python3.5/site-packages/google/cloud/datastore/batch.py:265: in commit
self._commit()
../../.local/lib/python3.5/site-packages/google/cloud/datastore/batch.py:242: in _commit
self.project, self._commit_request, self._id)
../../.local/lib/python3.5/site-packages/google/cloud/datastore/_http.py:627: in commit
response = self._datastore_api.commit(project, request)
../../.local/lib/python3.5/site-packages/google/cloud/datastore/_http.py:356: in commit
return self._stub.Commit(request_pb)
/usr/lib/python3.5/contextlib.py:77: in __exit__
self.gen.throw(type, value, traceback)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

@contextlib.contextmanager
def _grpc_catch_rendezvous():
    """Re-map gRPC exceptions that happen in context.

    .. _code.proto: https://github.com/googleapis/googleapis/blob/\
                    master/google/rpc/code.proto

    Remaps gRPC exceptions to the classes defined in
    :mod:`~google.cloud.exceptions` (according to the description
    in `code.proto`_).
    """
    try:
        yield
    except exceptions.GrpcRendezvous as exc:
        error_code = exc.code()
        error_class = _GRPC_ERROR_MAPPING.get(error_code)
        if error_class is None:
            raise
        else:
>               raise error_class(exc.details())
E               google.cloud.exceptions.ServiceUnavailable: 503 {"created":"@1488386955.558948202","description":"Secure read failed","file":"src/core/lib/security/transport/secure_endpoint.c","file_line":157,"grpc_status":14,"referenced_errors":[{"created":"@1488386955.558925887","description":"EOF","file":"src/core/lib/iomgr/tcp_posix.c","file_line":235}]}

../../.local/lib/python3.5/site-packages/google/cloud/datastore/_http.py:260: ServiceUnavailable

如何解决此问题？

google.cloud.exceptions.ServiceUnavailable：503并优化google datastore put_multi（）速度

0 个答案: