Question

以下是我同时执行请求的模式：

df2.sort_index(ascending=False, inplace=True)
df['closest_df2'] = df.index.map(lambda x: df2.index.asof(x))

df
Out[19]: 
      a  closest_df2
10    1           11
100   2          101
1000  3         1001

然而，这与每5,000个请求的错误rs = (grequests.get(url) for url in urls) res_items = grequests.map(rs) for num, res in enumerate(res_items): json_data = json.loads(res.text)崩溃。对于上述操作，更可靠的模式是什么 - 例如，如果单个请求不起作用，则重试网址最多五次？

Answer 1

这是一个选项，使用如下所述的指数退避：

def grequester(self, url, n=1):
    '''
    Google exponential backoff: https://developers.google.com/drive/web/handle-errors?hl=pt-pt
    '''
    MAX_TRIES = 8
    try:
        res = grequests.get(url)
    except:
        if n > MAX_TRIES:
            return None
        n += 1
        log.warning('Try #%s for %s...' % (n, url))
        time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0))  # add jitter 0-1000ms
        return self.grequester(url, n)
    else:
        return res

具有指数退避的请求

1 个答案: