Question

我的api中有以下POST方法。在向api发送大约350个条目的大型JSON集合之后，我收到了DeadlineExceedError异常。

我已经在使用ndb.multi_put_async了。我不知道还能做些什么来加快速度呢？

def post(self):
        arguments = self.reqparser.parse_args()
        json_records = arguments.get('records')
        user = User.query(User.email == request.authorization.username).get()
        if user:
            records_put_list = []
            events_put_list = []
            for json_record in json_records:
                record_id = json_record['record_id']
                rec = Record.get_or_insert(record_id,
                                           user=user.key,
                                           record_date=record_date,
                                           timestamp=record_timestamp)
                for json_event in json_record['events']:
                    event = Event.get_or_insert(json_event['event_id'],
                                                parent=rec.key,
                                                user=user.key,
                                                is_deleted=json_event['is_deleted'])
                    if event.timestamp < json_event['timestamp']:
                        event.user = user.key
                        event.record = rec.key
                        event.date_time = event_dt
                        event.timestamp = json_event['timestamp']
                        event.parent = rec.key
                        events_put_list.append(event)
            ndb.put_multi_async(records_put_list)
            ndb.put_multi_async(events_put_list)
            return '', 201
        else:
            return '', 401

这是例外。有什么建议我可以做什么？

   Traceback (most recent call last):
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
        result = handler(dict(self._environ), self._StartResponse)
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1836, in __call__
        return self.wsgi_app(environ, start_response)
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1817, in wsgi_app
        response = self.full_dispatch_request()
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1475, in full_dispatch_request
        rv = self.dispatch_request()
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/app.py", line 1461, in dispatch_request
        return self.view_functions[rule.endpoint](**req.view_args)
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask_restful/__init__.py", line 397, in wrapper
        resp = resource(*args, **kwargs)
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask/views.py", line 84, in view
        return self.dispatch_request(*args, **kwargs)
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/lib/flask_restful/__init__.py", line 487, in dispatch_request
        resp = meth(*args, **kwargs)
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/application/http_basic_auth.py", line 36, in decorated
        return f(*args, **kwargs)
      File "/base/data/home/apps/s~feeltracker1/1-0-1-0.376950929493492366/application/rest_api_view.py", line 362, in post
        is_deleted=json_event['is_deleted'])
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3401, in _get_or_insert
        return cls._get_or_insert_async(*args, **kwds).get_result()
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 325, in get_result
        self.check_success()
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 320, in check_success
        self.wait()
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 304, in wait
        if not ev.run1():
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 235, in run1
        delay = self.run0()
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 197, in run0
        callback(*args, **kwds)
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 474, in _on_future_completion
        self._help_tasklet_along(ns, ds_conn, gen, val)
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 371, in _help_tasklet_along
        value = gen.send(val)
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 1015, in transaction
        ok = yield tconn.async_commit(options)
    DeadlineExceededError

Answer 1

您没有使用正确的逻辑。不要将get_or_insert放在循环中 - 除了最少数量的记录之外，它永远不会起作用（并且通常是要避免的反模式）。而是创建实体并将它们附加到循环内的列表中。完成此操作后，put_multi将获取此列表并并行处理所有put。有一个问题是put_multi列表中有多少项是可以容忍的，但350远远不是我记得的限制。

Answer 2

您可以使用Task API将此处理作业拆分为较小的批次（例如，每个任务100个条目 - 您可以找到最佳数量）。

尽管使用了ndb.put_multi_async，但DeadlineExceededError

2 个答案: