Question

我正在尝试从http://www.qchp.org.qa/en/Pages/searchpractitioners.aspx中删除信息。我想做以下事情： - 从页面顶部的下拉列表中选择“Dentist” - 单击搜索 - 请注意，页面底部的信息会使用javascript动态更改 - 单击从业者姓名的超链接并显示一个弹出窗口 - 我想在每个从业者的json / csv文件中保存所有信息 - 我还想要在页面底部的其他页面上的信息，这些信息会更改保存div中的信息。

我对scrapy很新，只是研究了selenium，因为我读到了你需要selenium来获取动态信息的地方

所以我在scrapy应用程序中使用Selenium。不确定是否正确。我不知道最好的方法是什么。到目前为止，我有以下代码。我收到此错误sch_spider.py“，

line 21, in DmozSpider
    all_options = element.find_elements_by_tag_name("option")
NameError: name 'element' is not defined

sch_spider.py

from scrapy.spider import Spider
from scrapy.selector import Selector
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from scrapytutorial.items import SchItem
from selenium.webdriver.support.ui import Select

class DmozSpider(Spider):
    name = "sch"

    driver = webdriver.Firefox()
    driver.get("http://www.qchp.org.qa/en/Pages/searchpractitioners.aspx")
    select = Select(driver.find_element_by_name('ctl00$m$g_28bc0e11_4b8f_421f_84b7_d671de504bc3$ctl00$drp_practitionerType'))
    all_options = element.find_elements_by_tag_name("option")

    for option in all_options:
        if option.get_attribute("value") == "4":  #Dentist
            option.click()
        ends
        break

    driver.find_element_by_name("ctl00$m$g_28bc0e11_4b8f_421f_84b7_d671de504bc3$ctl00$Searchbtn").click()


    def parse(self, response):

        all_docs = element.find_elements_by_tag_name("td")
        for name in all_docs:
            name.click()
            alert = driver.switch_to_alert()
            sel = Selector(response)
            ma = sel.xpath('//table')
            items = []
            for site in ma:
                item = SchItem()
                item['name'] = site.xpath("//span[@id='PractitionerDetails1_lbl_Name']/text()").extract()
                item['profession'] = site.xpath("//span[@id='PractitionerDetails1_lbl_Profession']/text()").extract()
                item['scope_of_practise'] = site.xpath("//span[@id='PractitionerDetails1_lbl_sop']/text()").extract()
                item['instituition'] = site.xpath("//span[@id='PractitionerDetails1_lbl_institution']/text()").extract()
                item['license'] = site.xpath("//span[@id='PractitionerDetails1_lbl_LicenceNo']/text()").extract()
                item['license_expiry_date'] = site.xpath("//span[@id='PractitionerDetails1_lbl_LicenceExpiry']/text()").extract()
                item['qualification'] = site.xpath("//span[@id='PractitionerDetails1_lbl_Qualification']/text()").extract()

                items.append(item)
            return items

items.py

from scrapy.item import Item, Field

class SchItem(Item):

    name = Field()
    profession = Field()
    scope_of_practise = Field()
    instituition = Field()
    license = Field()
    license_expiry_date = Field()
    qualification = Field()

Answer 1

您不应该将以下代码中的element.find_elements ..更改为select.find_element ..

  select = Select(driver.find_element_by_name('ctl00$m$g_28bc0e11_4b8f_421f_84b7_d671de504bc3$ctl00$drp_practitionerType'))
  all_options = element.find_elements_by_tag_name("option")

或者更不应该使用select.options？

python scrapy：抓取动态信息

1 个答案: