Question

我编写了一个程序，该程序使用Beautiful Soup从Crunchbase中提取公司列表的资金信息，并将该信息导出为CSV文件。我什至将请求间隔了30秒，程序运行良好，直到今天-现在，我什至在没有收到HTTPError：Forbidden的情况下也无法发送一个请求。

我一直在阅读此书，并且人们制作了IP循环程序，因为看来Crunchbase一直在阻塞我的IP地址-即使我循环用户代理，我仍然会被阻塞。我什至尝试使用几个免费的VPN，但仍然遭到封锁。

import urllib.request
from bs4 import BeautifulSoup
import csv
import time
import random

user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 
Safari/537.36'
headers = {'User-Agent': user_agent, }

def scraper(url):
    return_list = []

    try:
        request = urllib.request.Request(url, None, headers)
        response = urllib.request.urlopen(request)
    except:
        return_list.append("No Crunchbase Page Found")
        return_list.append("No Crunchbase Page Found")
        print("Not found")
    else:
        data = response.read()
        soup = BeautifulSoup(data, "html.parser")

        try:
            funding_status = soup.find_all("span", class_= "component--field-formatter field-type-enum ng-star-inserted")[1].text
            return_list.append(funding_status)
        except:
            return_list.append("N/A")

        try:
            last_funding_type = soup.find("a", class_= "cb-link component--field-formatter field-type-enum ng-star-inserted").text
            if last_funding_type[:6] != "Series" and last_funding_type[:7] != "Venture" and last_funding_type[:4] != "Seed" and last_funding_type[:3] != "Pre" and last_funding_type[:5] != "Angel" and last_funding_type[:7] != "Private" and last_funding_type[:4] != "Debt" and last_funding_type[:11] != "Convertible" and last_funding_type[:5] != "Grant" and last_funding_type[:9] != "Corporate" and last_funding_type[:6] != "Equity" and last_funding_type[:7] != "Product" and last_funding_type[:9] != "Secondary" and last_funding_type[:4] != "Post" and last_funding_type[:3] != "Non" and last_funding_type[:7] != "Initial" and last_funding_type[:7] != "Funding":
                return_list.append("N/A")
            else:
                return_list.append(last_funding_type)
        except:
            return_list.append("N/A")

    return return_list


user_input = input("CSV File Name (e.g: myfile.csv): ")
user_input2 = input("New CSV file name (e.g: newfile.csv): ")
print()
scrape_file = open(user_input, "r", newline = '', encoding = "utf-8")
row_count = sum(1 for row in csv.reader(scrape_file))
scrape_file = open(user_input, "r", newline = '', encoding = "utf-8")

new_file = open(user_input2, "w", newline = '', encoding = "utf-8")
writer = csv.writer(new_file)
writer.writerow(["Company Name", "Description", "Website", "Founded", 
"Product Name", "Country", "Funding Status", "Last Funding Type"])

count = 0
for row in csv.reader(scrape_file):
    company_name = row[0]
    if company_name == "Company Name":
        continue

    count += 1
    print("Scraping company {} of {}".format(count, row_count))

    company_name = company_name.replace(",", "")
    company_name = company_name.replace("'", "")
    company_name = company_name.replace("-", " ")
    company_name = company_name.replace(".", " ")
    s = "-"
    join_name = s.join(company_name.lower().split())
    company_url = "https://www.crunchbase.com/organization/" + join_name

    writer.writerow([row[0], row[1], row[2], row[3], row[4], row[5], scraper(company_url)[0], scraper(company_url)[1]])
    time.sleep(random.randint(30, 40))
new_file.close()
print("Done! You can now open your file %s." % user_input2)

如果有人能为我指出如何将IP循环集成到该项目中，以便它发送来自不同IP地址的请求，我将不胜感激！我不打算为私人代理付费，但是我看到人们使用公共地址来支付费用。谢谢！

Answer 1

如果您想收到答复，则需要某种代理，例如squidproxy，付费私有代理或公共代理（如您提到的VPN）。没有其他办法了。您可以在发送到某个虚假IP的数据包中欺骗您的IP，但不会收到响应。如果您希望使用代理，我建议您使用优秀的requests库，因为它是许多进行Web抓取的人的首选工具，并且使用代理非常简单。示例如下：

import requests
proxies = {
  'http': 'http://10.10.1.10:3128', #this could be an public proxy address
  'https': 'http://10.10.1.10:1080',
}
requests.get("https://www.google.com",proxies=proxies)

，如果您希望循环浏览一系列公共代理，只需遍历整个代理即可处理异常，如下所示：

import requests
import logging
proxies = [{
  'http': 'http://10.10.1.10:3128', #this could be an public proxy address
  'https': 'http://10.10.1.10:1080',
},...]

for proxy in proxies:
    try:
        requests.get("https://www.google.com",proxies=proxies)
        break
    except Exception as e:
        logging.exception(e)
        continue

如何将IP循环集成到我的网络抓取程序中？我一直被Crunchbase封锁

1 个答案: