无法使用Selenium单击按钮

Question

根据 Cannot pull data from pantip.com ，我试图从pantip.com提取数据，包括所有评论和每个评论的回复。

我在获取每个评论的回复文本时遇到问题。我使用硒单击按钮以获取文本内容。但是，只有当我将页面滚动到按钮的位置时，它才起作用。

如果我不滚动，这是一个错误。

WebDriverException: unknown error: Element <a href="javascript:void(0)" class="reply see-more">...</a> is not clickable at point (518, 507). Other element would receive the click: <select class="dropdown-jump">...</select>
  (Session info: chrome=71.0.3578.98)
  (Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 10.0.17134 x86_64)

有什么方法可以获取内部数据吗？我需要滚动到按钮吗？ 请建议我。

import requests
import re
from bs4 import BeautifulSoup
from selenium import webdriver

chrome_path = r"C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe"
url='https://pantip.com/topic/38372443'
driver = webdriver.Chrome(chrome_path)
driver.get(url)

content=driver.page_source
soup=BeautifulSoup(content,'lxml')  

#Click all buttons
for div in soup.find_all("div", id = lambda value: value and value.startswith("reply-comment-")):
    xPath =  '''//*[@id="''' + str(div['id']) + '''"]/a''' 
    button = driver.find_element_by_xpath(xPath).click()


# Get all comments   
text = list()    
for div in soup.find_all("div", id = lambda value: value and value.startswith("comment-")):
    if len(str(div.text).strip()) > 1:
        text.append(str(div.text).strip())
driver.quit()

Answer 1

页面底部有一个固定的导航面板，因此当您尝试单击按钮时，实际上是单击该面板中的元素，这就是引发Exception的原因……您可能需要

• 滚动到所需按钮
• 向下滚动

• 单击按钮以查看回复

  from selenium.webdriver.common.keys import Keys
  
  for reply in driver.find_elements_by_xpath('//div[starts-with(@id, "reply-comment-")]/a'):
      driver.execute_script('arguments[0].scrollIntoView();', reply)
      reply.send_keys(Keys.DOWN)
      reply.click()
  

Answer 2

要点击每个评论按钮并提取您不需要的每个评论的回复文本，只需 Beautiful Soup 和 Selenium 。要实现此目的，您必须将所需元素 scroll 滚动到Viewport中，然后可以使用以下解决方案：

• 代码块：

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  options = webdriver.ChromeOptions()
  options.add_argument("start-maximized")
  options.add_argument("disable-infobars")
  options.add_argument("--disable-extensions")
  driver= webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://pantip.com/topic/38372443")
  comment_buttons = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.reply.see-more span.focus-txt")))
  for button in comment_buttons:
      driver.execute_script("return arguments[0].scrollIntoView(true);", button)
      button.click()
      print("Comment button clicked")
  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[class^='comment'][data-refcm^='comment'] div.display-post-story>a"))).get_attribute("innerHTML"))
  driver.quit()
  

• 控制台输出：

  Comment button clicked
  Comment button clicked
  Comment button clicked
  Comment button clicked
  https://www.instagram.com/clintbondad
  .
  .
  .
  

Answer 3

硒可以使用execute_script("xxxx")执行脚本，例如：

script = """function getComments(){var comments=new Array();a = $('div.display-post-story');for (var i=0;i<a.length;i++){comments.push(a[i].innerText)};return comments;}"""
comments = driver.execute_script(script)
# then u can deal with all comments.