Question

以下是我到目前为止从https://n.rivals.com/state_rankings/2021/alabama抓取到的代码。我希望代码通过替换阿拉巴马州的所有州来遍历该地址。理想情况下，我还希望能够更改年份以供将来使用。我如何定义url和year / state1怎么办？

from selenium import webdriver
from selenium.common.exceptions import TimeoutException

TIMEOUT = 5

driver = webdriver.Firefox()
driver.set_page_load_timeout(TIMEOUT)

url = f"https://n.rivals.com/state_rankings/{year}/{state1}"
year = "2021"
state1 = "alabama"

try:
    driver.get(url)
except TimeoutException:
    pass

first_names = driver.find_elements_by_class_name('first-name')
first_names = [name.text for name in first_names]

last_names = driver.find_elements_by_class_name('last-name')
last_names = [name.text for name in last_names]

for first, last in zip(first_names, last_names):
    print(first, last)

player_positions = driver.find_elements_by_class_name('pos')
player_positions = [position.text for position in player_positions]

for position in player_positions:
    print(position)

data = driver.find_elements_by_xpath('//div[@class="break-text ng-binding ng-scope"]')
for d in data:
    location, highschool = d.text.strip().split('\n')
    city, state = location.split(',')
    print(city)
    print(state)
    print(highschool)

commit_status = driver.find_elements_by_class_name('school-name')
commit_status = [commit.text for commit in commit_status]

for commit in commit_status:
    print(commit)

driver.close()

Answer 1

您必须先创建变量，然后才能像这样引用它们：

year = "2021"
state1 = "alabama"
url = f"https://n.rivals.com/state_rankings/{year}/{state1}"

然后要使其在许多州循环，您需要执行以下操作：

from selenium import webdriver
from selenium.common.exceptions import TimeoutException

TIMEOUT = 5

driver = webdriver.Firefox()
driver.set_page_load_timeout(TIMEOUT)

def rivals_scrape(state, year):
    url = f"https://n.rivals.com/state_rankings/{year}/{state}"

    try:
        driver.get(url)
    except TimeoutException:
        pass

    first_names = driver.find_elements_by_class_name('first-name')

    ... rest of code ...

    for commit in commit_status:
        print(commit)

states = ["alabama", "georgia","texas"]

for state in states:
    rivals_scrape(state, "2021")

driver.close()

如何使用Selenium遍历URL进行抓取

1 个答案: