Selenium Webdriver在抓取动态数字的网页时不返回任何内容

时间:2020-06-08 21:58:46

标签: python selenium web-scraping

from selenium import webdriver

chrome_path = r"C:\chromedriver_win32\chromedriver.exe"
# create new instance of chrome in incognito mode
browser = webdriver.Chrome(chrome_path)

# go to website of interest
browser.get("https://www.moneyweb.co.za/tools-and-data/click-a-company/SOL/")

# get the financial value
value_element = browser.find_elements_by_id('share-tradingdata-o')
values = [x.text for x in titles_element]

print(values)

上面的代码没有返回任何信息,而不是返回股价,这说明我在这里可能会缺少什么。我是这个网站的新手。请协助。

2 个答案:

答案 0 :(得分:0)

您应该使用Wait等待该元素加载。

这是使用显式等待的解决方案:

gradle dependencies

此操作最多等待10秒,然后抛出const SomeDiv = styled.div` background: blue; `; export default function App() { const Div = ({ children, onClick }) => { return <SomeDiv onClick={() => onClick("testing")}>{children}</SomeDiv>; }; function test(val) { console.log("I am"); console.log(val); } return ( <div className="App"> <SomeDiv onClick={() => console.log("Hello")}> <span>Hello</span> </SomeDiv> <Div onClick={val => test(val)}>test</Div> </div> ); } ,除非它发现要在10秒内返回的元素值。默认情况下,from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By chrome_path = r"C:\chromedriver_win32\chromedriver.exe" # create new instance of chrome in incognito mode browser = webdriver.Chrome(chrome_path) # go to website of interest browser.get("https://www.moneyweb.co.za/tools-and-data/click-a-company/SOL/") # wait for the element to load for 10 seconds try: # get the financial value when its populated to the page value_element = WebDriverWait(driver, 10).until( EC.text_to_be_present_in_element_value((By.ID, "share-tradingdata-o")) ) print(value_element .text) finally: # after 10 seconds, maybe give up? driver.quit() 每500毫秒调用一次TimeoutException,直到成功返回。

您可以根据情况不断增加等待时间。

有关等待的更多信息,请参阅Selenium Wait Docs

答案 1 :(得分:0)

等待带有文本的元素,请使用此xpath表达式//span[@id='share-tradingdata-o' and text()!=''],它将起作用。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_path = r"C:\chromedriver_win32\chromedriver.exe"
# create new instance of chrome in incognito mode
browser = webdriver.Chrome(chrome_path)

browser.get("https://www.moneyweb.co.za/tools-and-data/click-a-company/SOL/")
wait = WebDriverWait(browser, 20)
value_element = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//span[@id='share-tradingdata-o' and text()!='']")))
values = [x.text for x in value_element]
print(values)
browser.quit()