Question

我正在一个项目中，我从Indeed的搜索结果中抓取数据。我的代码可以正常工作，直到结果的第8页左右，然后出现此错误：

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//*[contains(concat( " ", @class, " " ), concat( " ", "accessible-contrast-color-location", " " ))]"}
  (Session info: chrome=83.0.4103.106)

此消息仅在第8页左右出现，但有时会在第15页左右出现。请让我知道我需要对代码进行哪些更改才能使其刮刮100页。谢谢！！这是我的代码：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.by import By
import time

driver = webdriver.Chrome("/Users/nzalle/Downloads/chromedriver")

page = 0
SearchTerm = input("Search Term: ")

LocationSearch = input("Location: ")


NumPages = 100

Data = []

for x in range(NumPages + 1):
    url = ('https://www.indeed.com/jobs?q=environmental&l=United+States&radius=50&start=' + str(page))
    driver.get(url)
    time.sleep(10)
    page += 10
    for jobs in WebDriverWait(driver, 10).until(ec.visibility_of_all_elements_located((By.CSS_SELECTOR, "div[data-tn-component='organicJob']"))):
        title = jobs.find_element_by_xpath("./h2/a").text

        location = jobs.find_element_by_xpath('.//*[contains(concat( " ", @class, " " ), concat( " ", "accessible-contrast-color-location", " " ))]').text or jobs.find_element_by_xpath('//*[@id="recJobLoc_e0ead35930e2f2cd"]').text

        try:
            name = jobs.find_element_by_xpath(".//span[@class='company']/a").text
        except:
            name = jobs.find_element_by_xpath(".//span[@class='company']").text

        Data.append(title.strip())
        Data.append(','.strip())
        Data.append(name.strip())
        Data.append(','.strip())
        Data.append(location.strip())
        Data.append("\n")
with open("ScrapeResults.txt", "w") as output:
    output.write(''.join([*Data]))

driver.close()

为什么我的代码在8页后会停止工作？

0 个答案: