从网站抓取网页无法点击下拉元素

时间:2021-06-11 21:11:27

标签: python-3.x selenium web-scraping selenium-chromedriver

我正在尝试从不同床垫尺寸的网站上获取价格。我一直在尝试选择下拉菜单的元素,它随机工作。问题是元素是 li 标签而不是选择标签选项,所以我不能使用 selenium 的内置类。有人能告诉我我在这里做错了什么吗?

这是网址和代码。

Site url

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
import pandas as pd

options = Options()
#options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--window-size=1366,768")
options.add_argument("--disable-notifications")
options.add_experimental_option('prefs', {'intl.accept_languages': 'en_GB'})
#options.headless = True
url ='https://thesleepcompany.in/product/mattress/smart-ortho/'

driver = webdriver.Chrome(options=options)
driver.get(url)

def click_dropdown():
    driver.find_element_by_class_name('dk-select').click() # dropdown menu to access options

opt=driver.find_element_by_class_name('dk-select-options').find_elements_by_tag_name('li') # different mattress sizes options list

l=[]

for i in range(1,len(opt)):
    click_dropdown()
    try:
        opt[i].click()
    except Exception as e:
        print(e)
        print('click not successfull')
    l.append(driver.find_element_by_class_name('woocommerce-variation-price').find_element_by_tag_name('bdi').text)
    WebDriverWait(driver, 5)

    #click_dropdown()
    print(1)

我得到的 for 循环的输出是

Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element click intercepted: Element <li class="dk-option  attached enabled" data-value="75%e2%80%b3-x-60%e2%80%b3-x-6%e2%80%b3" role="option" aria-selected="false" id="dk6-75%e2%80%b3-x-60%e2%80%b3-x-6%e2%80%b3">...</li> is not clickable at point (832, 601). Other element would receive the click: <tr>...</tr>
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: element not interactable
  (Session info: chrome=91.0.4472.101)

click not successfull
1
Message: stale element reference: element is not attached to the page document
  (Session info: chrome=91.0.4472.101)

click not successfull
1

0 个答案:

没有答案