我想抓取一个网站。我从网站上抓取了表格，但无法抓取其他页面。不幸的是，这些页面没有href。
当我单击其他页面时页面没有href。我尝试了.click（），但它没有按预期工作。我共享了用于按钮功能的html代码。如何抓取所有页面而不仅仅是一页。
<li class="paginate_button">
<a href="#" aria-controls="datatablezz" data-dt-idx="1" tabindex="0">1</a>
table = soup.find(
"table", attrs={"class": "table dataTable no-footer"}
)
tableBody = table.find("tbody")
self.rows = tableBody.find_all("tr")
def addParameters(self):
for row in self.rows:
cols = row.find_all("td")
cols = [sat.text.strip() for sat in cols]
self.data.append([sat for sat in cols if sat])
答案 0 :(得分：0)
您需要类似硒的方法。您可以确定页数，然后单击下一个按钮进行浏览。
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
d = webdriver.Chrome()
d.get('https://www.turksat.com.tr/tr/turksat-frekans-listesi')
buttons = WebDriverWait(d,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".paginate_button ")))
numPages = int(buttons[-2].text) #get last page number
#do something with page 1
if numPages > 1:
for page in range(2, numPages + 1):
WebDriverWait(d, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#datatablezz_next'))).click() #click next
#do something
答案 1 :(得分：0)
如果您使用Selenium和pandas会很容易获得表数据。但是，以我的方式，您将获得每个页表而不是整个表。请查看是否有帮助。加载页面，以便可以有page_source
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
import pandas as pd
import time
driver = webdriver.Chrome()
driver.get('https://www.turksat.com.tr/tr/turksat-frekans-listesi')
i=2
while i<13:
time.sleep(5)
html = driver.page_source
dfs = pd.read_html(html)
print(dfs[0])
WebDriverWait(driver,10).until(expected_conditions.element_to_be_clickable((By.XPATH,"//a[text()='" + str(i) + "']"))).click()
i = i + 1
如果您没有安装熊猫，也很容易安装pip install pandas
输出将如下所示。
Kanal Frekans Polarizasyon ... V-PID A-PID Uydu
0 2000'ler 11012 V - Dikey ... 120.0 220 T3A
1 24 HD 12310 V - Dikey ... 5055.0 5155 T4A
2 24 KITCHEN HD 12460 H - Yatay ... 2959.0 3059 T4A
3 24 TV 12298 V - Dikey ... 3100.0 3200 T4A
4 360 HD 12310 V - Dikey ... 5056.0 5156 T4A
5 360 TV 12298 V - Dikey ... 3101.0 3201 T4A
6 42 EN FM 12344 V - Dikey ... NaN 5818 T4A
7 60S 11012 V - Dikey ... 124.0 224 T3A
8 80S 11012 V - Dikey ... 122.0 222 T3A
9 90S 11012 V - Dikey ... 121.0 221 T3A
10 A HABER 11883 V - Dikey ... 1401.0 1501 T4A
11 A HABER 12053 H - Yatay ... 1301.0 1401 T4A
12 A HABER HD 12053 H - Yatay ... 1306.0 1406 T4A
13 A HABER RADYO 12053 H - Yatay ... NaN 1408 T4A
14 A NEWS HD 11998 V - Dikey ... 4905.0 5005 T4A
15 A PARA HD 11998 V - Dikey ... 4906.0 5006 T4A
16 A SPOR 12053 H - Yatay ... 1302.0 1402 T4A
17 A SPOR HD 12053 H - Yatay ... 1307.0 1407 T4A
18 A SPOR RADYO 12053 H - Yatay ... NaN 1409 T4A
19 A2 HD 11998 V - Dikey ... 4907.0 5007 T4A
20 A2 TV 12053 H - Yatay ... 1314.0 1414 T4A
21 ADA TV 12610 H - Yatay ... 8005.0 8105 T3A
22 AKILLI TV 12034 V - Dikey ... 131.0 231 T4A
23 AKİT TV HD 12685 H - Yatay ... 6800.0 6900 T3A
24 AKRA FM 11509 H - Yatay ... NaN 6116 T3A
25 AKSU TV 12641 H - Yatay ... 5506.0 5606 T3A
26 AL JAZEERA ARAB 12457 V - Dikey ... 5308.0 5408 T4A
27 AL JAZEERA ENGLISH 12457 V - Dikey ... 5307.0 5407 T4A
28 AL QURAN AL KAREEM TV 12729 H - Yatay ... 5006.0 5106 T3A
29 AL SUNNAH AL NABAWIYAH TV 12729 H - Yatay ... 5007.0 5107 T3A
30 ALEM FM 12298 V - Dikey ... NaN 3202 T4A
31 AMBIANCE 11012 V - Dikey ... 101.0 201 T3A
32 AMBIANCE 11012 V - Dikey ... 101.0 201 T3A
33 ANADOLU DERNEK TV 12559 V - Dikey ... 5406.0 5506 T3A
34 ANADOLU TV 11976 H - Yatay ... 151.0 251 T4A
35 ANADOLUDA VUSLAT TV 12514 H - Yatay ... 6906.0 7006 T3A
36 ANIMAL PLANET HD 12187 V - Dikey ... 102.0 202 T4A
37 ANTALYA RADYO 11096 H - Yatay ... NaN 1018 T3A
38 AROUND THE WORLD 11012 V - Dikey ... 101.0 268 T3A
39 AS TV (BURSA) 12685 V - Dikey ... 5805.0 5905 T3A
40 ATV 12053 H - Yatay ... 1300.0 1400 T4A
41 ATV AVRUPA 11883 V - Dikey ... 1400.0 1500 T4A
42 ATV HD 12053 H - Yatay ... 1305.0 1405 T4A
43 AV TV 12729 V - Dikey ... 6640.0 6740 T3A
44 AZADLIQ VOA AZERI 11976 H - Yatay ... NaN 271 T4A
45 BABY TV 11012 V - Dikey ... 176.0 276 T3A
46 BAROQUE 11012 V - Dikey ... 125.0 225 T3A
47 BAYRAK FM 12610 H - Yatay ... NaN 8111 T3A
48 BAYRAM FM 12344 V - Dikey ... NaN 5814 T4A
49 BBC EARTH 11012 V - Dikey ... 103.0 203 T3A
[50 rows x 9 columns]
Kanal Frekans Polarizasyon ... V-PID A-PID Uydu
0 BBC WORLD NEWS 11012 V - Dikey ... 115.0 215 T3A
1 BBC WORLD NEWS 11853 H - Yatay ... 3803.0 3903 T4A
2 beIN GURME 11180 H - Yatay ... 4457.0 4557 T3A
3 beIN HD 11675 V - Dikey ... 3319.0 3423 T3A
4 beIN IZ 11675 V - Dikey ... 3306.0 3406 T3A
5 beIN MOVIES ACTION 11180 H - Yatay ... 4458.0 4558 T3A
6 beIN MOVIES ACTION HD 12072 V - Dikey ... 4965.0 5065 T4A
7 beIN MOVIES FAMILY 11675 V - Dikey ... 3316.0 3416 T3A
8 beIN MOVIES PREMIER HD 12072 V - Dikey ... 4964.0 5064 T4A
9 beIN MOVIES PREMIERE 11180 H - Yatay ... 4461.0 4561 T3A
10 beIN MOVIES STARS 11675 V - Dikey ... 3313.0 3413 T3A
11 beIN MOVIES TÜRK 11180 H - Yatay ... 4456.0 4556 T3A
12 beIN SERIES COMEDY 12072 V - Dikey ... 4962.0 5062 T4A
13 beIN SERIES DRAMA 12072 V - Dikey ... 4960.0 5060 T4A
14 beIN SERIES SCI-FI 12072 V - Dikey ... 4963.0 5063 T4A
15 beIN SERIES VICE 12072 V - Dikey ... 4961.0 5061 T4A
16 beIN SPORT 1 HD 11675 V - Dikey ... 3311.0 3411 T3A
17 beIN SPORT 1 HD 11675 V - Dikey ... 3311.0 3411 T3A
18 beIN SPORT 4 11675 V - Dikey ... 3307.0 3407 T3A
19 beIN SPORTS 1 11180 H - Yatay ... 4460.0 4560 T3A
20 beIN SPORTS 2 11180 H - Yatay ... 4462.0 4564 T3A
21 beIN SPORTS 3 11180 H - Yatay ... 4463.0 4566 T3A
22 beIN SPORTS HABER 12378 H - Yatay ... 6421.0 6521 T4A
23 beIN SPORTS HABER HD 12378 H - Yatay ... 6420.0 6520 T4A
24 beIN SPORTS MAX 1 12072 V - Dikey ... 4955.0 5055 T4A
25 beIN SPORTS MAX 2 12072 V - Dikey ... 4956.0 5056 T4A
26 BENGÜ TÜRK 12422 V - Dikey ... 5101.0 5201 T4A
27 BERAT TV 11916 V - Dikey ... 6307.0 6407 T4A
28 BERAT TV 12344 V - Dikey ... 5700.0 5800 T4A
29 BERAT TV 12422 H - Yatay ... 5203.0 5303 T4A
30 BEREKET AGRİ 12379 V - Dikey ... 6206.0 6306 T4A
31 BEREKET TV 12457 V - Dikey ... 5302.0 5402 T4A
32 BEST FM 12034 V - Dikey ... NaN 277 T4A
33 BEYAZ TV 12379 V - Dikey ... 6200.0 6300 T4A
34 BEYAZ TV HD 12379 V - Dikey ... 6201.0 6301 T4A
35 BEYKENT TV 12729 V - Dikey ... 6639.0 6739 T3A
36 BJK TV 11675 V - Dikey ... 3318.0 3422 T3A
37 BLOOMBERG HT 11053 H - Yatay ... 2126.0 2226 T3A
38 BLOOMBERG HT HD 12209 H - Yatay ... 2002.0 2102 T4A
39 BOOMERANG TV HD 12187 V - Dikey ... 116.0 216 T4A
40 BROADWAY 11012 V - Dikey ... 134.0 234 T3A
41 BRT 1 12610 H - Yatay ... 8000.0 8100 T3A
42 BRT 2 12610 H - Yatay ... 8001.0 8101 T3A
43 BRT HD 12610 H - Yatay ... 8002.0 8102 T3A
44 BRT RADYO 1 12610 H - Yatay ... NaN 8110 T3A
45 BRTV 12264 V - Dikey ... 3415.0 3515 T4A
46 BURSA TV 12264 V - Dikey ... 3414.0 3514 T4A
47 CANLAR TV 11061 H - Yatay ... 3077.0 3177 T3A
48 CARTON NETWORK 11976 H - Yatay ... 115.0 215 T4A
49 CHANSONS 11012 V - Dikey ... 101.0 249 T3A
[50 rows x 9 columns]
Kanal Frekans Polarizasyon ... V-PID A-PID Uydu
0 CHILLOUT 11012 V - Dikey ... 101.0 228 T3A
1 CLASSIC MASTERS 11012 V - Dikey ... 127.0 227 T3A
2 CLASSICAL CALM 11012 V - Dikey ... 101.0 258 T3A
3 CLASSICAL GREATS 11012 V - Dikey ... 101.0 233 T3A
4 CLASSICAL ORCHESTRAL 11012 V - Dikey ... 101.0 232 T3A
5 CNN INTERNATIONAL 11012 V - Dikey ... 108.0 208 T3A
6 CNN TURK HD 12245 H - Yatay ... 102.0 202 T4A
7 CNN TÜRK 12034 V - Dikey ... 101.0 201 T4A
8 CNNTURK RADYO 12034 V - Dikey ... NaN 271 T4A
9 COOL JAZZ 11012 V - Dikey ... 101.0 230 T3A
10 DA VINCI LEARNING 11012 V - Dikey ... 104.0 204 T3A
11 DA VINCI LEARNING 11789 H - Yatay ... 4660.0 4760 T4A
12 DANCE CLUBBIN 11012 V - Dikey ... 165.0 265 T3A
13 DEEP BLUES 11012 V - Dikey ... 101.0 235 T3A
14 DEHA TV 12264 V - Dikey ... 3409.0 3509 T4A
15 DIS YAY ORTA DALGA ARAPÇA 11096 H - Yatay ... NaN 1316 T3A
16 DISCOVERY CHANNEL 12378 H - Yatay ... 6412.0 6512 T4A
17 DISCOVERY CHANNEL HD 12187 V - Dikey ... 101.0 201 T4A
18 DISNEY CHANNEL 12422 H - Yatay ... 5200.0 5300 T4A
19 DISNEY JUNIOR 11880 H - Yatay ... 4130.0 4230 T4A
20 DiYANET RADYO 12379 V - Dikey ... NaN 6312 T4A
21 DİYALOG TV 12610 H - Yatay ... 8003.0 8103 T3A
22 DİYANET RİSALET RADYO 12379 V - Dikey ... NaN 6216 T4A
23 DİYANET TV 12422 H - Yatay ... 5204.0 5304 T4A
24 DİYANET TV HD 12379 V - Dikey ... 6203.0 6303 T4A
25 DİYAR TV 12685 V - Dikey ... 5814.0 5914 T3A
26 DİZİ PREMIUM 12187 V - Dikey ... 109.0 209 T4A
27 DİZİ SMART MAX 12187 V - Dikey ... 114.0 214 T4A
28 DMAX 12015 H - Yatay ... 1802.0 1902 T4A
29 DMAX HD 12015 H - Yatay ... 1815.0 1915 T4A
30 DOCUBOX TV 11012 V - Dikey ... 106.0 206 T3A
31 DOST FM 12685 V - Dikey ... NaN 5941 T3A
32 DOST TV 12685 V - Dikey ... 5822.0 5922 T3A
33 DREAM TV 12034 V - Dikey ... 105.0 205 T4A
34 DREAM TÜRK 12034 V - Dikey ... 107.0 207 T4A
35 DREAM TÜRK HD 12245 H - Yatay ... 106.0 206 T4A
36 DRT DENiZLi 12729 V - Dikey ... 6631.0 6731 T3A
37 DURU TV 12685 V - Dikey ... 5812.0 5912 T3A
38 EASY LISTENING 11012 V - Dikey ... 101.0 257 T3A
39 EDESSA TV 12685 V - Dikey ... 5809.0 5909 T3A
40 EGE TÜRK TV 12685 V - Dikey ... 5816.0 5916 T3A
41 EKİN TÜRK TV 11916 V - Dikey ... 6308.0 6408 T4A
42 EKOTÜRK 12245 H - Yatay ... 115.0 215 T4A
43 ELTR TV 11021 H - Yatay ... 4194.0 4373 T3A
44 ENGLISH CLUB TV HD 11789 H - Yatay ... 4659.0 4759 T4A
45 ER TV 12264 V - Dikey ... 3412.0 3512 T4A
46 ERKAM RADYO 11509 H - Yatay ... NaN 6115 T3A
47 ERZURUM RADYO 11096 H - Yatay ... NaN 1216 T3A
48 ES TV 12605 V - Dikey ... 5604.0 5704 T3A
49 ES TV 12685 V - Dikey ... 5801.0 5901 T3A
[50 rows x 9 columns]
答案 2 :(得分：0)
您无需逐页浏览，所有数据都带有单个请求，并且使用JavaScript进行表格渲染。您应该使用下面的选择器来获取所有频道数据：
div.modal-content
您应该解析所需的数据，例如：
channels = soup.select("div.modal-content")
print(str(len(channels)))
for channel in channels:
title = channel.select_one(".modal-title").text.strip()
...