我是Web Scraping的新手,遇到了这个问题。我尝试单击“下一步”按钮,但Selenium告诉我它不可单击,因为另一个元素将其遮盖了。我尝试使用“等待”,但它也不能解决问题。
我怀疑它可能与scrape_each_artist()函数有关,因为当我注释掉它时,Selenium可以单击“下一步”。我的scrape_each_artist()函数仅使用BeautifulSoup和Requests从每个艺术家的个人资料中抓取数据。
driver = webdriver.Firefox()
driver.get('https://www.saffronart.com/Artist/ArtistList.aspx/')
num_of_pages = int(driver.find_element_by_id('ContentPlaceHolder1_PagingLinks1_rptPagingLinks_lbPagingFooter').text)
#get the number of pages
for page in range(num_of_pages): #go through all the pages
profile_list = driver.find_elements_by_link_text('Profile')
for profile in profile_list: #go through every artist profile in the page
profile_link = profile.get_attribute('href')
scrape_each_artist(profile_link)
try:
next_page = WebDriverWait(driver, 3).until(EC.element_to_be_clickable((By.ID, 'ContentPlaceHolder1_lnkNext')))
#wait 3 seconds until the 'Next' button appears
except TimeoutException:
print ("Loading took too much time!")
next_page.click()
driver.close()
更新:我已经通过向下滚动页面解决了该问题。这是我所做的更改:
driver.execute_script("scroll(0, 1050)")
next_page = driver.find_element_by_id('ContentPlaceHolder1_lnkNext')
next_page.click()
谢谢!