从具有selenium和python的optgroup中选择一个选项

时间：2018-05-07 15:55:02

标签： python selenium web-crawler

我想从这个optgroup中选择一个值，然后给我一个链接下拉列表。

<div class="searchbar">
    <select id="q" multiple="" tabindex="-1" class="select2-hidden-accessible" aria-hidden="true">
    <option></option>
    <option class="q-all-text" value="al:all">Search all text</option>

    <optgroup label="Business Type">
        <option value="bt:Buyer">Buyer</option>
        <option value="bt:Farmer/Rancher">Farmer/Rancher</option>
        <option value="bt:Farmers Market">Farmers Market</option>
        <option value="bt:Fishery">Fishery</option>
        <option value="bt:Food Bank">Food Bank</option>
    </optgroup>

到目前为止，这是我的代码：

driver = webdriver.Chrome('/path/to/chromedriver')
driver.get("https://foodmarketmaker.com/main/mmsearch")
iframe = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(iframe)

select  = Select(driver.find_element_by_class_name("select2-hidden- accessible"))
select.select_by_value("bt:Farmer/Rancher")
links = driver.find_elements_by_tag_name('a')
print(links)
for link in links:
    print(link.get_attribute('href'))

我得到一个异常，该元素不存在，或者当我通过索引访问时，由于aria-hidden属性为true，我得到一个ElementNotVisibleException。有没有办法解决这个问题？

2 个答案:

答案 0 :(得分：1)

from selenium import webdriver as web from selenium.webdriver.support.ui import WebDriverWait as wait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By driver = web.Chrome('/path/to/chromedriver') driver.get("https://foodmarketmaker.com/main/mmsearch") iframe = driver.find_element_by_tag_name("iframe") driver.switch_to.frame(iframe) wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//section[@id="search-right"]//input[@placeholder="start typing to search"]'))).click() wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//li[.="Bass"]'))).click() links = wait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//section[@id="search-results"]//a[.//*[name()="svg"]]'))) for link in links: print(link.get_attribute('href')) 不是您需要处理的节点，因为它不可见。

尝试使用以下代码获取所需的输出：

li

P.S。您需要将'//li[.="Bass"]' XPath中spark_config = SparkConf().setAppName(application_name) ss = SparkSession.builder.config(conf=spark_config).getOrCreate() class StringAccumulatorParam(AccumulatorParam): def zero(self, v): return [] def addInPlace(self, variable, value): variable.append(value) return variable errorCount = ss.sparkContext.accumulator(0) errorValues = ss.sparkContext.accumulator("", StringAccumulatorParam()) newSchema = StructType([ StructField("id", IntegerType(), True), StructField("name", StringType(), True) StructField("status", BooleanType(), True)]) errorDF = ss.read.json("/Users/test.jsonl") errorDF2 = ss.createDataFrame(errorDF, newSchema).cache() def checkErrorCount(row): global errorCount errorDF2["id"] = row. newSchema["id"] errorCount.add(1) errorValues.add(errorDF2["id"]) errorDF.foreach(lambda x: checkErrorCount(x)) print("{} rows had questionable values.".format(errorCount.value)) ss.stop() 节点的文本值替换为必需选项

答案 1 :(得分：0)

根据您共享的HTML打印所有选项，您必须诱导 WebDriverWait 切换到所需的框架，然后单击搜索框以获取< em> options 可见如下：

  • 代码块：

    from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://foodmarketmaker.com/main/mmsearch")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@src='https://search.foodmarketmaker.com']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='searchbar']//input[@class='select2-search__field' and @placeholder='start typing to search']"))).click()
links = driver.find_elements_by_xpath("//li[@class='select2-results__option' and @aria-label='Business Type']/strong//following::ul[1]//li")
for link in links:
    print(link.get_attribute('innerHTML'))

  • 控制台输出：

    Buyer
Buyer: Baked Goods
Buyer: Beverages
Buyer: Dairy
Buyer: Fish &amp; Seafood
Buyer: Fruit &amp; Nuts
Buyer: Grains &amp; OilSeed
Buyer: Herbs
Buyer: Meat &amp; Poultry
Buyer: Miscellaneous
Buyer: Specialty Products
Buyer: Sugar/Confectionary &amp; Snacks
Buyer: Vegetables
Eating &amp; Drinking Place
Eating &amp; Drinking Place: Cafeteria &amp; Buffet
Eating &amp; Drinking Place: Caterer
Eating &amp; Drinking Place: Drinking Place
Eating &amp; Drinking Place: Food Service Contractor
Eating &amp; Drinking Place: Mobile Food Service
Eating &amp; Drinking Place: Restaurant
Eating &amp; Drinking Place: Snack &amp; Nonalcoholic Beverage Bar
Farmer/Rancher
Farmer/Rancher: Dairy
Farmer/Rancher: Forage
Farmer/Rancher: Fruits &amp; Nuts
Farmer/Rancher: Grains
Farmer/Rancher: Herbs
Farmer/Rancher: Livestock Production (Beef Cattle)
Farmer/Rancher: Meat &amp; Poultry
Farmer/Rancher: Specialty Products
Farmer/Rancher: Vegetables
Farmers Market
Fishery
Fishery: Fish/Shellfish/Seafood
Food Bank
Food Bank: Food Bank
Food Hub
Food Retailer
Food Retailer: Baked Goods Store
Food Retailer: Beer, Wine &amp; Liquor Store
Food Retailer: Candy, Nut &amp; Confectionery
Food Retailer: Fish Market
Food Retailer: Fruit &amp; Vegetable Market
Food Retailer: Grocery/Convenience Store
Food Retailer: Health Supplement Store
Food Retailer: Meat &amp; Poultry Market
Food Retailer: Other Direct Selling Establishments
Food Retailer: Other Specialty Food Store
General Member
Processor/Packing Shed
Processor/Packing Shed: Animal Food Manufacturing
Processor/Packing Shed: Bakery Products
Processor/Packing Shed: Beverages
Processor/Packing Shed: Dairy Products
Processor/Packing Shed: Fish/Shellfish/Seafood Products
Processor/Packing Shed: Fruit &amp; Vegetable Products
Processor/Packing Shed: Grain &amp; Oilseed Milling
Processor/Packing Shed: Meat &amp; Poultry Products
Processor/Packing Shed: Miscellaneous
Processor/Packing Shed: Packing Shed
Processor/Packing Shed: Sugar and Confectionery Products
Tourism
Tourism: Agritourism
Tourism: Fishing Charter
Tourism: Hunting
Wholesaler
Wholesaler: Beer, Wine &amp; Alcoholic Beverages
Wholesaler: Confectionery &amp; Snacks
Wholesaler: Dairy Products
Wholesaler: Farm Products
Wholesaler: Fish/Shellfish/Seafood
Wholesaler: Fruit &amp; Vegetable
Wholesaler: General Grocery
Wholesaler: Meat &amp; Poultry
Wholesaler: Other Grocery
Wholesaler: Packaged Frozen Food
Winery
Winery: Blends or Generic Wine
Winery: Fortified Wine
Winery: Fruit Wine
Winery: Other Wine Products
Winery: Red Wine
Winery: Sparkling Wine
Winery: White Wine
相关问题
最新问题