无法周期性地单击“更多”按钮以获取所有完整评论

Question

我已经在python中创建了一个与硒结合使用的脚本，以从Google地图的特定页面获取所有评论。该页面上有很多评论，只有在该页面向下滚动时，它们才可见。我的脚本可以成功完成所有任务。

但是，我目前面临的唯一问题是某些评论带有More按钮，该按钮可以单击以显示完整评论。

其中之一是这样：

website address

我尝试过：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://www.google.com/maps/place/Pizzeria+Di+Matteo/@40.8512552,14.255779,17z/data=!4m7!3m6!1s0x133b0841ef6e38e5:0xece6ea09987e9baf!8m2!3d40.8512512!4d14.2579677!9m1!1b1"

driver = webdriver.Chrome()
driver.get(link)
wait = WebDriverWait(driver,10)

while True:
    try:
        elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "[class='section-loading-spinner']")))
        driver.execute_script("arguments[0].scrollIntoView();",elem)
    except Exception:
        break

    for see_more in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "button[class^='section-expand-review']"))):
        see_more.click()


for item in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".section-review-content"))):
    name = item.find_element_by_css_selector("[class='section-review-title'] > span").text
    try:
        review = item.find_element_by_css_selector("[class='section-review-text']").text
    except AttributeError:
        review = ""
    print(name)

driver.quit()

当前，以上脚本在碰到stale element这一行时会引发for see_more in wait.until().click()错误。

如何周期性地单击该More按钮以获取所有完整评论？

Answer 1

如果使用WebdriverWait和presence_of_all_elements_located，则它会在给定时间内等待搜索该元素，如果该元素未附加到html，则会收到错误消息。

但是，请检查网页中存在的元素的长度，然后单击该元素。

if len(driver.find_elements_by_css_selector("button[class^='section-expand-review']"))>0:
  driver.find_element_by_css_selector("button[class^='section-expand-review']").click()

这是代码。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://www.google.com/maps/place/Ecstasy/@23.7399982,90.3732109,17z/data=!3m1!4b1!4m7!3m6!1s0x3755b8caa669d5e3:0x41f47ddcc39a556e!8m2!3d23.7399933!4d90.3753996!9m1!1b1"

driver = webdriver.Chrome()
driver.get(link)
wait = WebDriverWait(driver,10)

while True:
    try:
        elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "[class='section-loading-spinner']")))
        driver.execute_script("arguments[0].scrollIntoView();",elem)
    except Exception:
        break

    if len(driver.find_elements_by_css_selector("button[class^='section-expand-review']"))>0:
        driver.find_element_by_css_selector("button[class^='section-expand-review']").click()
        print('pass')



for item in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".section-review-content"))):
    name = item.find_element_by_css_selector("[class='section-review-title'] > span").text
    try:
        review = item.find_element_by_css_selector("[class='section-review-text']").text
    except AttributeError:
        review = ""
    print(name)

driver.quit()

已编辑

 if len(driver.find_elements_by_css_selector("button[class^='section-expand-review']"))>0:

    for item in driver.find_elements_by_css_selector("button[class^='section-expand-review']"):
        item.location_once_scrolled_into_view
        item.click()
        time.sleep(2)

Answer 2

这是和我一起工作的：- 您可以将其放入 for 循环或您的方法中以获取所有评论。

try:                
   driver.find_element_by_class_name("mapsConsumerUiSubviewSectionReview__section-expand-review").click()
except:
   continue