Question

我在我的应用中设置了四个队列。

queue:
- name: tasks-stage-0
  rate: 475/s
  retry_parameters:
    task_retry_limit: 5
    min_backoff_seconds: 5
    max_backoff_seconds: 100
    max_doublings: 5
  bucket_size: 100

- name: tasks-stage-1
  rate: 475/s
  retry_parameters:
    task_retry_limit: 5
    min_backoff_seconds: 5
    max_backoff_seconds: 100
    max_doublings: 5
  bucket_size: 100

- name: tasks-stage-2     <------- !!!!
  rate: 4/s
  retry_parameters:
    task_retry_limit: 5
    min_backoff_seconds: 5
    max_backoff_seconds: 100
    max_doublings: 5
  bucket_size: 100

- name: tasks-stage-3
  rate: 475/s
  retry_parameters:
    task_retry_limit: 5
    min_backoff_seconds: 5
    max_backoff_seconds: 100
    max_doublings: 5
  bucket_size: 100

tasks_stage_2是对API进行外部调用的一个队列。该API每秒限制4次。

在运行抓取时，我从该队列中收到了相当多的错误消息：

ConnectionError: ('Connection aborted.', HTTPException('Deadline exceeded while waiting for HTTP response from URL: xx

是不是因为我每秒击中API的次数超过4次？我的设置是否正确？或者为什么我的截止日期超出错误？

更新

taskqueue.Task(url='/worker/stage-2', headers=header, payload=json.dumps({'news_url_string':news.key.urlsafe()})).add(queue_name='tasks-stage-2')

更新2：

Exception on /worker/stage-2 [POST]
Traceback (most recent call last):
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/flask_restful/__init__.py", line 263, in error_router
    return original_handler(e)
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/flask/views.py", line 149, in dispatch_request
    return meth(*args, **kwargs)
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/news/worker/stage2.py", line 16, in post
    OpenCalais.generate_tags_for_news(news_url_string)
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/news/opencalais/opencalais.py", line 31, in generate_tags_for_news
    response.raise_for_status()
  File "/base/data/home/apps/s~news-prod/1-0-10.382350631067599473/lib/requests/models.py", line 831, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: Internal Server Error

Answer 1

您的队列配置似乎有些偏差（请参阅底部的参考资料以更好地了解每个值的作用）

- name: tasks-stage-2
  rate: 4/s  # Once per second, add 4 new tokens to the bucket
  max_concurrent_requests: 4  # No more than 4 at a time!
  retry_parameters:
    task_retry_limit: 5
    min_backoff_seconds: 5
    max_backoff_seconds: 100
    max_doublings: 5
  bucket_size: 4  # Only for places each time it's checked

参考文献：

如何设置像信号量一样的队列？

1 个答案: