Question

我不明白为什么这段代码没有进入解析方法。它与doc：http://doc.scrapy.org/en/latest/topics/spiders.html中的基本蜘蛛示例非常相似而且我很确定这在当天早些时候有效...不确定我是否修改过某些东西..

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from scrapy.spider import Spider
from scrapy.selector import HtmlXPathSelector
from scrapy import log
from scrapy.selector import Selector


class jobSpider(Spider):
    name='jobSpider'
    wd = webdriver.Chrome()
    wd.get("some url")
    wd.switch_to_frame("cible")


def parse(self, response):
    log.start()

    wait = WebDriverWait(wd, 10).until(
    (EC.visibility_of_element_located((By.ID, 'blocResultat'))))
    print(wd.page_source)

    stuff=Selector(text=wd.page_source).xpath('//a[contains(@onclick,"win=window.o    pen(\'JobOffs")]').extract() 
    print(stuff)

Answer 1

您的parse(self, response):方法不属于jobSpider类。如果您查看Scrapy documentation，您会发现parse方法需要成为您的蜘蛛类的一种方法。

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from scrapy.spider import Spider
from scrapy.selector import HtmlXPathSelector
from scrapy import log
from scrapy.selector import Selector


class jobSpider(Spider):
    name='jobSpider'
    wd = webdriver.Chrome()
    wd.get("https://www.leforem.be/particuliers/offres-emploi-recherche-par-criteres.html?exParfullText=&exPar_search_=true&exParGeographyEdi=true")
    wd.switch_to_frame("cible")


    def parse(self, response):
        log.start()

        wait = WebDriverWait(wd, 10).until(
        (EC.visibility_of_element_located((By.ID, 'blocResultat'))))
        print(wd.page_source)

        stuff=Selector(text=wd.page_source).xpath('//a[contains(@onclick,"win=window.o    pen(\'JobOffs")]').extract() 
        print(stuff)

此外，您需要在parse方法中引用类数据，方法是在类中的任何数据上使用self.前缀。

此外，您还缺少蜘蛛上的start_urls列表。没有它，蜘蛛不知道从哪里开始，也什么都不做。

Answer 2

您必须使用self访问属性，而parse是该类的一部分。

def parse(self, response):
    log.start()
    wait = WebDriverWait(self.wd, 10).until(
    (EC.visibility_of_element_located((By.ID, 'blocResultat'))))
    print(self.wd.page_source)
    stuff=Selector(text=self.wd.page_source).xpath('//a[contains(@onclick,"win=window.o    pen(\'JobOffs")]').extract() 
    print(stuff)

Scrapy没有进入解析方法

2 个答案: