Question

我正在尝试使用Python和Selenium来抓取网站并保存信息。刮擦很简单，只需要在两个下拉菜单中选择州和区，单击提交按钮，然后读取和写入表到csv。

我确信我的软件包安装正确，我的程序工作正常，但只有部分时间。我的猜测是，如果没有合适的Selenium驱动程序等等，我的程序会崩溃，因为它无法找到正确的css_selector。我将在下面发布该程序，如果有人对如何正确合并Selenium驱动程序等等提出任何建议，我将非常感谢您的帮助。

非常感谢，以及这个程序：

import time
import re
import string
import urllib.parse
import pandas
import numpy
import os
import csv

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

driver = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
url = "https://myhpgas.in/myHPGas/HPGas/LocateDistributor.aspx"
driver.set_window_size(1120, 550)
driver.get(url);
time.sleep(5) 
stateList = driver.find_element_by_css_selector("#ContentPlaceHolder1_ddlState")
options = stateList.find_elements_by_tag_name("option")

optionsList = []

for option in options:
    optionsList.append(option.get_attribute("value"))

optionsList[1:len(optionsList)]
for optionValue in optionsList:

    select = Select(driver.find_element_by_css_selector("#ContentPlaceHolder1_ddlState"))
    select.select_by_value(optionValue)
    districtList = driver.find_element_by_css_selector("#ContentPlaceHolder1_ddlDistrict")
    distOptions = districtList.find_elements_by_tag_name("option")

    distOptionsList = []

    for distOption in distOptions: #iterate over the options, place attribute value in list
        distOptionsList.append(distOption.get_attribute("value"))

    for distOptionValue in distOptionsList[1:len(distOptionsList)]:

        distSelect = Select(driver.find_element_by_css_selector("#ContentPlaceHolder1_ddlDistrict"))
        distSelect.select_by_value(distOptionValue)
        driver.find_element_by_css_selector('#ContentPlaceHolder1_btnShowList').click()
        data = []                      
        for tr in driver.find_elements_by_css_selector('#ContentPlaceHolder1_gvDistributor'):
            tds = tr.find_elements_by_tag_name('td')
            if tds: 
                data.append([td.text for td in tds])
        print(data)
        dataRows = int(numpy.array(data).size / 7)
        rowsTimesColumns = (dataRows * 7) -1
        newArray = numpy.array(data)
        outArray = newArray[0:rowsTimesColumns]
        test = pandas.DataFrame(outArray.reshape(dataRows,7), columns=['no', 'distributor', 'address','contact1', 'contact2', 'contact3', 'map'])

        file_path = 'Users/outpath' + '_' + optionValue + '_' + distOptionValue + '.csv'
        test.to_csv(file_path, sep=',')
        driver.back()
    driver.back()

Answer 1

你能告诉我哪一行返回错误！？还有如何使用xpaths？？

我看不到实现显式等待的语句

WebDriverWait(driver, 30).until(EC.presence_of_element_located_by(By.CSS_SELECTOR,*your css selector*))

如何为我的Selenium抓取程序添加等待？

1 个答案: