我正在尝试从此网站http://sekolah.data.kemdikbud.go.id/抓取学校数据
我似乎无法找出一种方法来单击下一页以从其他页面加载条目。单击下一页时,URL不会更改。不同页面的HTML代码如下:
<a href="javascript:void(0)" onclick="paging(2)">2</a>
这是我的代码:
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time
option = webdriver.ChromeOptions()
option.add_argument('--incognito')
webdriver = "/Users/rainer/Desktop/learnpython/web/chromedriver"
driver = Chrome(executable_path=webdriver, chrome_options=option)
url="http://sekolah.data.kemdikbud.go.id/"
driver.get(url)
WebDriverWait(driver,15).until(EC.element_to_be_clickable((By.XPATH,"//button[text()='Cari Sekolah']"))).click()
for i in range(2,15):
try:
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.XPATH, "//ul[@id='list']//a[text()='" + str(i) + "']"))).click()
print("page number #" + str(i))
time.sleep(1)
except:
print("No such element")
break
答案 0 :(得分:1)
我执行了您的代码,它似乎正常工作。页面的URL不会更改,但是页面上显示的学校实际上会更改。
看看,我添加了代码以打印出学校名称:
url="http://sekolah.data.kemdikbud.go.id/"
driver.get(url)
wait = WebDriverWait(driver,15)
wait.until(EC.element_to_be_clickable((By.XPATH,"//button[text()='Cari Sekolah']"))).click()
for i in range(2, 15):
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//ul[@id='list']//a[text()='" + str(i) + "']"))).click()
print("page number #" + str(i))
# Print school names to make sure they change per page
names = driver.find_elements(By.CSS_SELECTOR, 'ul > li.list-group-item:first-child a')
print([name.text for name in names])
time.sleep(2)
except Exception as e:
print(e)
break
输出:
page number #2
['(69839180) KELOMPOK BERMAIN KUSUMA HATI', '(K5660869) LKP ALVY', '(20103743) SMKS PGRI 1 JAKARTA', '(20269763) TK MARDI WALUYA']
page number #3
['(69990549) KB GENERASI CERDAS', '(69953341) KB HARAPAN BUNDA', '(P2965122) PKBM TELAGO', '(69930497) SPS ANGGREK']
page number #4
['(69877674) KB WADAH KASIH', '(20567521) TK AISYIYAH BUSTANUL ATHFAL', '(30406867) TK AISYIYAH BUSTANUL ATHFAL 2', '(69734039) TK PGRI TINGGAR']
...
因此,看起来您仍然可以抓取数据,并且一切正常。
希望这会有所帮助。如果您的代码输出不同,请告诉我。祝你好运。