Scrapy不会爬行任何东西

时间:2019-11-16 14:15:24

标签: python selenium scrapy

When I am running it in terminal the output is enter image description here,当我在终端中运行代码时,Scrapy Crawler正在爬行,但是当我在编辑器中运行相同代码时,它就无法工作。我的代码是:

from scrapy import Spider
from selenium import webdriver
from scrapy.selector import Selector
from scrapy.http import Request
class BooksSpider(Spider):
    name = 'books'
    allowed_domains = ['books.toscrape.com']


def start_request(self, response):
    self.driver=webdriver.Chrome(r'C:\Users\Night-Watch\Desktop\chromedriver.exe')
    self.driver.get('http://books.toscrape.com')
    sel=Selector(text=self.driver.page_source)


    books=sel.xpath('//h3/a/@href').extract()
    print(books)

1 个答案:

答案 0 :(得分:0)

  1. 您确定要使用Selenium抓取简单的测试站点吗?这是相当大的开销。
  2. 您能否提供错误文本和堆栈跟踪信息以了解情况的全部情况?