Question

我正尝试从具有多级下拉菜单的网站上抓取数据，每次选择一个项目时，它都会更改子下拉菜单的子项目。问题在于，对于每个循环，它都从下拉项中提取相同的子项。选择发生，但不会代表循环中的新选择更新项目有谁能帮助我为什么我没有得到理想的结果。也许是因为我的下拉列表是在Java Script之类的东西中。

例如下图中的此菜单： image of the dropdown 我走了这么远：

enter code here

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
import csv

//#from selenium.webdriver.support import Select 
import time

print ("opening chorome....")  
driver = webdriver.Chrome()
driver.get('https://www.wheelmax.com/')
time.sleep(10)

csvData = ['Year', 'Make', 'Model', 'Body', 'Submodel', 'Size']

//#variables
yeart = []
make= []
model=[]
body = []
submodel = []
size = []
Yindex = Mkindex = Mdindex = Bdindex = Smindex = Sindex = 0

print ("waiting for program to set variables....")
time.sleep(20)

print ("initializing and setting variables....")

//#initializing Year
Year = Select(driver.find_element_by_id("icm-years-select"))
Year.select_by_value('2020')
yr = driver.find_elements(By.XPATH, '//*[@id="icm-years-select"]')
time.sleep(15)

//#initializing Make
Make = Select(driver.find_element_by_id("icm-makes-select"))
Make.select_by_index(1)
mk = driver.find_elements(By.XPATH, '//*[@id="icm-makes-select"]')
time.sleep(15)

//#initializing Model
Model = Select(driver.find_element_by_id("icm-models-select"))
Model.select_by_index(1)
mdl = driver.find_elements(By.XPATH, '//*[@id="icm-models-select"]')
time.sleep(15)

//#initializing body
Body = Select(driver.find_element_by_id("icm-drivebodies-select"))
Body.select_by_index(1)
bdy = driver.find_elements(By.XPATH, '//*[@id="icm-drivebodies-select"]')
time.sleep(15)

//#initializing submodel
Submodel = Select(driver.find_element_by_id("icm-submodels-select"))
Submodel.select_by_index(1)
sbm = driver.find_elements(By.XPATH, '//*[@id="icm-submodels-select"]')
time.sleep(15)

//#initializing size
Size = Select(driver.find_element_by_id("icm-sizes-select"))
Size.select_by_index(0)
siz = driver.find_elements(By.XPATH, '//*[@id="icm-sizes-select"]')
time.sleep(5)


Cyr = Cmk = Cmd = Cbd = Csmd = Csz = ""

print ("fetching data from variables....")

for y in yr:
    obj1 = driver.find_element_by_id("icm-years-select")
    Year = Select(obj1)
    Year.select_by_index(++Yindex)
    obj1.click()
    #obj1.click()
    yeart.append(y.text)
    Cyr = y.text
    time.sleep(10)
    for m in mk:
        obj2 = driver.find_element_by_id("icm-makes-select")
        Make = Select(obj2)
        Make.select_by_index(++Mkindex)
        obj2.click()
        #obj2.click()
        make.append(m.text)
        Cmk = m.text
        time.sleep(10)
        for md in mdl:
            Mdindex =0
            obj3 = driver.find_element_by_id("icm-models-select")
            Model = Select(obj3)
            Model.select_by_index(++Mdindex)
            obj3.click()
            #obj3.click(clickobj)
            model.append(md.text)
            Cmd = md.text
            time.sleep(10)
            Bdindex = 0
            for bd in bdy:
                obj4 = driver.find_element_by_id("icm-drivebodies-select")
                Body = Select(obj4)
                Body.select_by_index(++Bdindex)
                obj4.click()
                #obj4.click(clickobj2)
                body.append(bd.text)
                Cbd = bd.text
                time.sleep(10)
                Smindex = 0
                for sm in sbm:
                    obj5 = driver.find_element_by_id("icm-submodels-select")
                    Submodel = Select(obj5)
                    obj5.click()
                    Submodel.select_by_index(++Smindex)
                    #obj5.click(clickobj5)
                    submodel.append(sm.text)
                    Csmd = sm.text
                    time.sleep(10)
                    Sindex = 0
                    for sz in siz:
                        Size = Select(driver.find_element_by_id("icm-sizes-select"))
                        Size.select_by_index(++Sindex)
                        size.append(sz.text)
                        Scz = sz.text
                        csvData += [Cyr, Cmk, Cmd, Cbd,Csmd, Csz]

Answer 1

我想您无法用漂亮的汤解析年份的原因是因为当所有漂亮的汤下载页面时，包含所有年份的'select'标签的'select'标签尚不存在/被隐藏了。通过执行我假设的其他JavaScript将其添加到DOM。如果使用浏览器的开发人员工具（例如，用于Mozilla的F12）查看已加载页面的DOM，则会看到包含要查找的信息的标记为<select id="icm-years-select"">。如果您尝试使用下载了漂亮汤的对象来解析该标签，则会得到一个空的标签对象列表：

from bs4 import BeautifulSoup
from requests import get
response = get('https://www.wheelmax.com/')
yourSoup = BeautifulSoup(response.text, "lxml")
print(len(yourSoup.select('div #vehicle-search'))) // length = 1 -> visible
print()
print(len(yourSoup.select('#icm-years-select')))    // length = 0 -> not visible

因此，如果您想通过所有方式使用Python来获得使用年限，我想您可能会尝试单击相应的标签，然后使用请求/美丽的汤/或硒模块的某种组合再次解析，这将需要一个多挖一点：-）

否则，如果您只是很快需要解析的年份，请使用JavaScript：

countYears = document.getElementById('icm-years-select').length;
yearArray = [];
for (i = 0; i < countYears; i++) {yearArray.push(document.getElementById('icm-years-select')[i].value)};

Answer 2

由于https://www.wheelmax.com具有相互依赖的多级下拉菜单，例如，如果您选择Select Year下拉选项，则在基于Select Make的选定年份下启用并显示根据所选的年份选项。

因此，基本上，您需要使用Selenium包来处理动态选项。

根据浏览器安装硒Web驱动程序

下载chrome网络驱动程序：

http://chromedriver.chromium.org/downloads

为Chrome浏览器安装Web驱动程序：

unzip ~/Downloads/chromedriver_linux64.zip -d ~/Downloads
chmod +x ~/Downloads/chromedriver
sudo mv -f ~/Downloads/chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver

硒教程

https://selenium-python.readthedocs.io/

例如使用硒来选择多个下拉选项

from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time

driver = webdriver.Chrome()
driver.get('https://www.wheelmax.com/')
time.sleep(4)

selectYear = Select(driver.find_element_by_id("icm-years-select"))
selectYear.select_by_value('2019')

time.sleep(2)

selectMakes = Select(driver.find_element_by_id("icm-makes-select"))
selectMakes.select_by_value('58')

更新：

选择下拉选项值或统计选项总数

for option in selectYear.options:
    print(option.text)

print(len(selectYear.options))

Se more

Answer 3

如何使用python beautifulsoup从下拉菜单中提取数据

页面执行回调以填充年份。简单地模仿那个。

如果您实际上需要更改年份并从相关下拉列表中进行选择（这将成为另一个问题），则需要使用浏览器自动化功能，例如硒，或手动执行此操作并检查“网络”标签，以查看是否存在您可以模仿以提交选择的xhr请求。

import requests

r = requests.get('https://www.iconfigurators.com/json2/?returnType=json&bypass=true&id=13898&callback=yearObj').json()
years = [item['year'] for item in r['years']]
print(years)

如何使用python beautifulsoup从下拉菜单中提取数据

3 个答案: