Question

我已根据urllib3.util.retry和here的建议，使用here实现了requests会话的重试机制。

现在，我正在尝试找出添加将在每次重试尝试时调用的回调函数的最佳方法是什么。

为进一步说明自己，如果Retry对象或请求get方法都可以添加回调函数，那就太好了。也许像这样：

import requests
from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def retry_callback(url):
    print url   

s = requests.Session()
retries = Retry(total=5, status_forcelist=[ 500, 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))

url = 'http://httpstat.us/500'
s.get(url, callback=retry_callback, callback_params=[url])

我知道可以使用日志记录来打印url，但这只是更复杂用途的一个简单示例。
如果这不是最好的python编码，请原谅，但我希望它足够清楚。

谢谢。

Answer 1

您可以继承Retry类以添加该功能。

对于给定的连接尝试，这是与Retry实例的完整交互流：

Retry.increment()。 Retry.is_retry()方法返回true。
- .increment()将重新引发该错误（如果存在），并且该对象已配置为不重试该特定类别的错误。
- .increment()调用Retry.new()以创建更新后的实例，并更新所有相关计数器，并用新的RequestHistory() instance（命名元组）修改history属性。
- Retry.is_exhausted()将引发Retry.new()异常。当它跟踪的任何计数器降到0以下（忽略设置为is_exhausted()的计数器）时，None返回true。
- .increment()返回新的Retry实例。
返回值Retry.increment()替换了跟踪的旧Retry实例。如果存在重定向，则调用Retry.sleep_for_retry()（如果有Retry-After头则处于睡眠状态），否则调用Retry.sleep()（调用self.sleep_for_retry()以兑现{{1 }}标头，否则，如果有退避策略，则仅在休眠状态下使用）。然后，使用新的Retry-After实例进行递归连接调用。

这给您3个不错的回调点；在Retry的开头，创建新的.increment()实例时，以及在Retry周围的上下文管理器中，让回调否决异常或在退出时更新返回的重试策略。

这是super().increment()开头的样子：

.increment()

请注意，import logging logger = getLogger(__name__) class CallbackRetry(Retry): def __init__(self, *args, **kwargs): self._callback = kwargs.pop('callback', None) super(CallbackRetry, self).__init__(*args, **kwargs) def new(self, **kw): # pass along the subclass additional information when creating # a new instance. kw['callback'] = self._callback return super(CallbackRetry, self).new(**kw) def increment(self, method, url, *args, **kwargs): if self._callback: try: self._callback(url) except Exception: logger.exception('Callback raised an exception, ignoring') return super(CallbackRetry, self).increment(method, url, *args, **kwargs)参数实际上只是 URL路径，请求的净位置部分被省略了（您必须从url中提取出来）参数，它具有_pool，.scheme和.host属性。

演示：

.port

在>>> def retry_callback(url): ... print('Callback invoked with', url) ... >>> s = requests.Session() >>> retries = CallbackRetry(total=5, status_forcelist=[500, 502, 503, 504], callback=retry_callback) >>> s.mount('http://', HTTPAdapter(max_retries=retries)) >>> s.get('http://httpstat.us/500') Callback invoked with /500 Callback invoked with /500 Callback invoked with /500 Callback invoked with /500 Callback invoked with /500 Callback invoked with /500 Traceback (most recent call last): File "/.../lib/python3.6/site-packages/requests/adapters.py", line 440, in send timeout=timeout File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen body_pos=body_pos, **response_kw) File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen body_pos=body_pos, **response_kw) File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen body_pos=body_pos, **response_kw) [Previous line repeated 1 more times] File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 712, in urlopen retries = retries.increment(method, url, response=response, _pool=self) File "<stdin>", line 8, in increment File "/.../lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/.../lib/python3.6/site-packages/requests/sessions.py", line 521, in get return self.request('GET', url, **kwargs) File "/.../lib/python3.6/site-packages/requests/sessions.py", line 508, in request resp = self.send(prep, **send_kwargs) File "/.../lib/python3.6/site-packages/requests/sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "/.../lib/python3.6/site-packages/requests/adapters.py", line 499, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses',))方法中添加一个钩子将使您可以为下一次尝试调整策略，并让您自省.new()属性，但不能避免异常引发

使用request / urllib3在每次重试尝试中添加回调函数

1 个答案: