Question

我有一个用户ID列表，我对抓取他们的信誉很感兴趣。

我使用beautifulsoup编写了一个脚本，该脚本可抓取用户的声誉。但是问题是，当我的脚本运行不到一分钟时，出现请求过多错误。之后，我也无法在浏览器上手动打开堆栈溢出。

我的问题是，如何在不出现过多请求错误的情况下抓取信誉？

我的代码如下：

for id in df['target']:
    url='https://stackoverflow.com/users/'+str(id)
    print(url)
    response=get(url)
    html_soup=BeautifulSoup(response.text, 'html.parser') 
    site_title = html_soup.find("title").contents[0]
    if "Page Not Found - Stack Overflow" in site_title:
        reputation="NA"
    else:    
        reputation=(html_soup.find(class_='grid--cell fs-title fc-dark')).contents[0].replace(',', "")
        print(reputation)

Answer 1

我建议使用Python time模块并在您的for循环中抛出time.sleep(5)。该错误是由于您在很短的时间内发出太多请求而引起的。不过，您可能需要考虑实际的睡眠时间才能正确使用。

Answer 2

您可以检查response.status_code == 429，并查看响应中是否有一个值告诉您要等待多长时间，然后等待被要求的秒数。

我在这里重复了这个问题。我在内容或标题中找不到任何等待时间的信息。

我建议您放一些油门并进行调整，直到您对结果满意为止。

有关从Stack Exchange Data Explorer获取用户信誉的示例，请参见https://data.stackexchange.com/stackoverflow/query/952/top-500-answerers-on-the-site。

示例如下。

#!/usr/bin/env python

import time
import requests
from bs4 import BeautifulSoup

df={}
df['target']=[ ... ] # see https://data.stackexchange.com/stackoverflow/query/952/top-500-answerers-on-the-site

throttle = 2
whoa = 450

with open('results.txt','w') as file_handler:
    file_handler.write('url\treputation\n')
    for id in df['target']:
        time.sleep(throttle)
        url='https://stackoverflow.com/users/'+str(id)
        print(url)
        response=requests.get(url)
        while response.status_code == 429:
            print(response.content)
            print(response.headers)
            time.sleep(whoa)
            response=requests.get(url)
        html_soup=BeautifulSoup(response.text, 'html.parser')
        site_title = html_soup.find("title").contents[0]
        if "Page Not Found - Stack Overflow" in site_title:
            reputation="NA"
        else:    
            reputation=(html_soup.find(class_='grid--cell fs-title fc-dark')).contents[0].replace(',', "")
        print('reputation: %s' % reputation)
        file_handler.write('%s\t%s\n' % (url,reputation))

示例错误内容。

<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>Too Many Requests - Stack Exchange</title>
    <style type="text/css">
        body
        {
            color: #333;
            font-family: 'Helvetica Neue', Arial, sans-serif;
            font-size: 14px;
            background: #fff url('img/bg-noise.png') repeat left top;
            line-height: 1.4;
        }
        h1
        {
            font-size: 170%;
            line-height: 34px;
            font-weight: normal;
        }
        a { color: #366fb3; }
        a:visited { color: #12457c; }
        .wrapper {
            width:960px;
            margin: 100px auto;
            text-align:left;
        }
        .msg {
            float: left;
            width: 700px;
            padding-top: 18px;
            margin-left: 18px;
        }
    </style>
</head>
<body>
    <div class="wrapper">
        <div style="float: left;">
            <img src="https://cdn.sstatic.net/stackexchange/img/apple-touch-icon.png" alt="Stack Exchange" />
        </div>
        <div class="msg">
            <h1>Too many requests</h1>
                        <p>This IP address (nnn.nnn.nnn.nnn) has performed an unusual high number of requests and has been temporarily rate limited. If you believe this to be in error, please contact us at <a href="mailto:team@stackexchange.com?Subject=Rate%20limiting%20of%20nnn.nnn.nnn.nnn%20(Request%20ID%3A%202158483152-SYD)">team@stackexchange.com</a>.</p>
                        <p>When contacting us, please include the following information in the email:</p>
                        <p>Method: rate limit</p>
                        <p>XID: 2158483152-SYD</p>
                        <p>IP: nnn.nnn.nnn.nnn</p>
                        <p>X-Forwarded-For: nnn.nnn.nnn.nnn</p>
                        <p>User-Agent: python-requests/2.20.1</p>
                        <p>Reason: Request rate.</p>
                        <p>Time: Tue, 20 Nov 2018 21:10:55 GMT</p>
                        <p>URL: stackoverflow.com/users/nnnnnnn</p>
                        <p>Browser Location: <span id="jslocation">(not loaded)</span></p>
        </div>
    </div>
    <script>document.getElementById('jslocation').innerHTML = window.location.href;</script>
</body>
</html>

错误标头示例。

{ “内容长度”：“ 2054”， “ Via”：“ 1.1清漆”， “ X-Cache”：“ MISS”， “ X-DNS-Prefetch-Control”：“关闭”， “接受范围”：“字节”， “ X-Timer”：“ S1542748255.394076，VS0，VE0”， “服务器”：“清漆”， “重试”：“ 0”， “连接”：“关闭”， “ X-Served-By”：“ cache-syd18924-SYD”， “ X-Cache-Hits”：“ 0”， “ Date”：“星期二，2018年11月20日格林尼治标准时间”， “ Content-Type”：“ text / html” }

从Stack Overflow抓取用户信誉时出现太多请求错误

2 个答案: