Question

我有以下可以正常工作的网络爬虫脚本，我需要的是一种在每个请求中集成身份验证或发送 cookie 的方法

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class TheFriendlyNeighbourhoodSpider(CrawlSpider):
    name = 'TheFriendlyNeighbourhoodSpider'    
    allowed_domains = ['one.google.com']
    start_urls = ['https://one.google.com/about']   
    custom_settings = {
    'LOG_LEVEL': 'INFO'
    }
    rules = (
        Rule(LinkExtractor(), callback='parse_item', follow=True),
    )
    def parse_item(self, response):
        print(response.url)

Answer 1

request_with_cookies = Request(url="http://www.example.com",
                           cookies=[{'name': 'currency',
                                    'value': 'USD',
                                    'domain': 'example.com',
                                    'path': '/currency'}])

更多信息：https://docs.scrapy.org/en/latest/topics/request-response.html

带有身份验证或 cookie 的 Python 抓取

1 个答案: