尝试抓取网站时连接拒绝错误

时间:2019-05-29 15:55:19

标签: python scrapy splash scrapy-splash connection-refused

尝试运行Web抓取程序,但我不断收到错误“另一方拒绝连接:111:拒绝连接”。我认为这可能与我的Splash_url有关,即使我已验证localhost8050已启动并正在运行。因为如果不是那样,那么我不知道可能是什么问题。

setting.py

BOT_NAME = 'scrapy_javascript'

SPIDER_MODULES = ['scrapy_javascript.spiders']
NEWSPIDER_MODULE = 'scrapy_javascript.spiders'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# The path of the csv file that contains the pairs
PROXY_CSV_FILE = "proxies.csv"

DOWNLOADER_MIDDLEWARES = {
        'scrapy_splash.SplashCookiesMiddleware': 723,
        'scrapy_splash.SplashMiddleware': 725,
        'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPLASH_URL = 'http://localhost:8050'
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

SplashSpider.py

import csv
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import GameItem


# process the csv file so the url + ip address + useragent pairs are the same as defined in the file
# returns a list of dictionaries, example:
# [ {'url': 'http://www.starcitygames.com/catalog/category/Rivals%20of%20Ixalan',
#    'ip': 'http://204.152.114.244:8050',
#    'ua': "Mozilla/5.0 (BlackBerry; U; BlackBerry 9320; en-GB) AppleWebKit/534.11"},
#    ...
# ]
def process_csv(csv_file):
    data = []
    reader = csv.reader(csv_file)
    next(reader)
    for fields in reader:
        if fields[0] != "":
            url = fields[0]
        else:
            continue # skip the whole row if the url column is empty
        if fields[1] != "":
            ip = "http://" + fields[1] + ":8050" # adding http and port because this is the needed scheme
        if fields[2] != "":
            useragent = fields[2]
        data.append({"url": url, "ip": ip, "ua": useragent})
    return data


class MySpider(Spider):
    name = 'splash_spider'  # Name of Spider

    # notice that we don't need to define start_urls
    # just make sure to get all the urls you want to scrape inside start_requests function

    # getting all the url + ip address + useragent pairs then request them
    def start_requests(self):

        # get the file path of the csv file that contains the pairs rom the settings.py
        with open(self.settings["PROXY_CSV_FILE"], mode="r") as csv_file:
            # requests is a list of dictionaries like this -> {url: str, ua: str, ip: str}
            requests = process_csv(csv_file)

        for req in requests:
            # no need to create custom middlewares
            # just pass useragent using the headers param, and pass proxy using the meta param
            yield SplashRequest(url=req["url"], callback=self.parse, args={"wait": 3},
                                headers={"User-Agent": req["ua"]},
                                meta={"proxy": (req["ip"])})

    # Scraping
    def parse(self, response):
        item = GameItem()
        for game in response.css("tr"):
            # Card Name
            item["card_name"] = game.css("a.card_popup::text").extract_first()
            yield item

错误消息:

Traceback (most recent call last):
  File "/home/trno224/.local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
2019-05-28 11:59:44 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.starcitygames.com/catalog/category/Avacyn%20Restored via http://localhost:8050/render.html> (failed 3 times): Connection was refused by other side: 111: Connection refused.
2019-05-28 11:59:44 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.starcitygames.com/catalog/category/Commander%202013%20Edition via http://localhost:8050/render.html> (failed 3 times): Connection was refused by other side: 111: Connection refused.
2019-05-28 11:59:44 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://www.starcitygames.com/catalog/category/Weatherlight>: HTTP status code is not handled or not allowed
2019-05-28 11:59:44 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.starcitygames.com/catalog/category/Archenemy%20Nicol%20Bolas via http://localhost:8050/render.html> (failed 3 times): Connection was refused by other side: 111: Connection refused.
2019-05-28 11:59:44 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://www.starcitygames.com/catalog/category/Welcome%20Deck%202016 via http://localhost:8050/render.html> (referer: None)
2019-05-28 11:59:44 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.starcitygames.com/catalog/category/Avacyn%20Restored via http://localhost:8050/render.html>
2019-05-28 21:06:32 [scrapy.core.engine] INFO: Closing spider (finished)
2019-05-28 21:06:32 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 558,
'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 558,
'downloader/request_bytes': 387967,
'downloader/request_count': 560,
'downloader/request_method_count/GET': 2,
'downloader/request_method_count/POST': 558,
'downloader/response_bytes': 1344,
'downloader/response_count': 2,
'downloader/response_status_count/200': 1,
'downloader/response_status_count/404': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 5, 29, 1, 6, 32, 860344),
'log_count/DEBUG': 560,
'log_count/ERROR': 186,
'log_count/INFO': 33,
'memusage/max': 58634240,
'memusage/startup': 54370304,
'response_received_count': 2,
'retry/count': 372,
'retry/max_reached': 186,
'retry/reason_count/twisted.internet.error.ConnectionRefusedError': 372,
'robotstxt/request_count': 2,
'robotstxt/response_count': 2,
'robotstxt/response_status_count/200': 1,
'robotstxt/response_status_count/404': 1,
'scheduler/dequeued': 744,
'scheduler/dequeued/memory': 744,
'scheduler/enqueued': 744,
'scheduler/enqueued/memory': 744,
'splash/render.html/request_count': 186,
'start_time': datetime.datetime(2019, 5, 29, 0, 42, 2, 24066)}
2019-05-28 21:06:32 [scrapy.core.engine] INFO: Spider closed (finished)

1 个答案:

答案 0 :(得分:0)

为了解决问题,我需要将splash_url设置为等于当前代理,以防止程序崩溃。

splash_url = req["ip"],