Question

我在这里使用python scrapy with selenium我添加了代码

import scrapy
from examble.items import exambleItem
from selenium import webdriver


class exambleSpider(scrapy.Spider):
    name = "examble"
    allowed_domain = ['examble.com']
    start_urls = ['http://www.examble.com/kids/page-1.html']

    def parse(self,response):
        driver = webdriver.Chrome()
        driver.get(response)
        for i in range(1,36):
            item = ExambleItem()
            item['Name'] = 'sdds'       
            #driver = webdriver.Chrome()
            #driver.get(response)
            xpath = "//*[@id='wrapper']/main/div/div[2]/table/tbody/tr[%d]/td[1]/a" % i
            elem = driver.find_element_by_xpath(xpath)
            elem.click()
            yield item

在此我用scrapy抓取运行代码类型错误发生错误是TypeError＆lt; 200 http://www.examble.com/kids/page-1.html＆gt;不是JSON Serializable

我正在用driver.get替换driver.get（响应）（＆＃39; http://www.examble.com/kids/page-1.html＆＃39;）它运行的应该是确切的错误

Answer 1

response是scrapy.Response个对象。你在这里看到的是response.url，因为driver.get()需要一个字符串参数。

只需将response替换为response.url：

driver.get(response)

在Scrapy with Selenium中，类型错误url不是JSON可序列化的

1 个答案: