网站：https://chp.coth.com/entries
我对BS4有一些经验，但这使我陷入了循环。按下按钮后，DataTables通过函数调用发出XHR请求，但是使用开发人员工具中的“网络”选项卡将POST显示为“ https://chp.coth.com/entries/List”。
我注意到from数据包含页码，并且我确定有一种方法可以使用request发送带有JSON字符串的帖子，直到列表结尾，但是我不确定如何去做它。没有Selenium或类似工具，这可能吗？
答案 0 :(得分：1)
此脚本遍历所有页面（1..7，您可以增加此数字以解析更多结果），并向https://chp.coth.com/entries/List发出POST请求。重要的部分是将页眉'X-Requested-With'设置为'XMLHttpRequest'和正确的页码：
import requests
from bs4 import BeautifulSoup
url = 'https://chp.coth.com/entries/List'
data = {
'num': '',
'page': '1',
'sort_on': 'number',
'sort_type': 'ASC',
}
headers = {'X-Requested-With' : 'XMLHttpRequest'}
for page_num in range(1, 8):
data['page'] = page_num
r = requests.post(url, params=data, headers=headers)
soup = BeautifulSoup(r.text, 'lxml')
for tr in soup.select('tr'):
for i, td in enumerate(tr.select('td')):
if i == 0:
print('{: >5}'.format(td.text), end='|')
else:
print('{: >35}'.format(td.text), end='|')
print()
打印：
501| LEADGURIA| WESLEY FOX| LAURIE GRAYSON|
502| WINCHESTER| CHIARA SASSANO| LAURIE GRAYSON|
503| MERLIN'S TANGO| SYONA MORRISON| LAURIE GRAYSON|
504| GOLDINO VAN KOEKSHOF| COLLEEN GILMORE| LAURIE GRAYSON|
510| CRESPO VDL| KAREN BANISTER| KAREN BANISTER|
511| HERRADURA| KAREN BANISTER| KAREN BANISTER|
512| SUGAR RUSH| HALEY HONEGGER| ALEXIA HONEGGER|
520| BONJORNO| ELISE VANDAMME, DENISE TILLEY| DENISE TILLEY|
521| SIR RIO| CHELSEA MAGNESS, STACEY SWINK| LAURA BAKER|
522| CANDY MAN| CHELSEA MAGNESS, STACEY SWINK| LAURA BAKER|
523| SERGEANT PEPPER| CHELSEA MAGNESS, STACEY SWINK| LAURA BAKER|
524| CAYENNE F| STACEY SWINK, BRUCE BANKER| LAURA BAKER|
525| DESMOND| STACEY SWINK| LAURA BAKER|
526| NIKO'S TAKURA| ISABELLA MANCINI| LAURA BAKER|
527| THE BIG EASY| JESSICA OAKLIEF| LAURA BAKER|
528| ZEPHYR’S SONG| ELEANOR ROZANCE| LAURA BAKER|
529| CALISTO| LUCIA WELLSO| LAURA BAKER|
530| CHESHIRE JACK| REVEN SWANSON| LAURA BAKER|
531| KEEPSAKE| STACEY SWINK| LAURA BAKER|
532| GRACEFUL| LAURA BAKER| LAURA BAKER|
535| BULL RUN'S AGAPE| BRYN SADLER| WENDY HAIG|
536| CLIFF AFFAIR| BRYN SADLER| WENDY HAIG|
537| CINVEN CR| BRYN SADLER| WENDY HAIG|
538| CEZARRO| BRYN SADLER| WENDY HAIG|
539| BULL RUN'S LIVING| BRYN SADLER| WENDY HAIG|
540| CHICAGO VAN OVERIS Z| BRYN SADLER| WENDY HAIG|
546| ALL I EVER WANTED| TEGAN ZIMMERMAN| RENAE COATES|
547| COASTAL CZONE|TEGAN ZIMMERMAN, RENAE COATES, KAITLIN CAMPBELL| RENAE COATES|
554| CALLIGRAPHY| AVA UHRIG| ALEXIA HONEGGER|
555| UTWO DE LA CERISAIE| ISABELLA UHRIG| ALEXIA HONEGGER|
556| MR CAPTAIN COOK B| ALEXIA HONEGGER, PAIGE MILES| ALEXIA HONEGGER|
558| KIRIKOU DES BRIMBELLES| JAYDEN SCOTT| ALEXIA HONEGGER|
560| INDOKTRO K VAN 'T KATTENHEYE| SUSAN GRIFFIS| SUSAN GRIFFIS|
561| DEAUVILLE VAN OVERIS Z| SUSAN GRIFFIS| SUSAN GRIFFIS|
563| TRISTAN| ERIC LAMON, GRETCHEN AYCOCK| GRETCHEN AYCOCK|
564| TRILOGY| GRETCHEN AYCOCK| GRETCHEN AYCOCK|
566| EMPIRE BEAR BONES| RACHEL MALONEY| RACHEL MALONEY|
567| WINGS SUBLIEME| RACHEL MALONEY| RACHEL MALONEY|
571| J.E.S. CIBELLE| ASHLEY BUCHANAN| JOHN MC CONNELL|
572| CHIANTI CLASSICO| ASHLEY BUCHANAN| JOHN MC CONNELL|
573| POLAR LIGHT| ASHLEY BUCHANAN| JOHN MC CONNELL|
574| BYKARLOS| KATHY KRAMER| ALEXIA HONEGGER|
576| VONNOH| SABRINA OHLROGGE| ALEXIA HONEGGER|
577| JUST SAY SO| HANNAH WRIGHT| LAURIE GRAYSON|
580| BURGUNDY B| AMELIA JOHNSON| ALEXIA HONEGGER|
583| PITCH PERFECT| EMILY SCHOELZEL| REBECCA JOHNSON|
584| CATANA S| SHELBY BURLESON| AMY HENZE|
585| CALLE 87| ASHLEY RHEINGOLD, SARAH WATSON| AMY HENZE|
586| AMEDEO| JULIA FISCHER| AMY HENZE|
589| ITON| ISABELLA LOMBARDO| REBECCA JOHNSON|
590| PLEASANT VALLEY SUNDAY| SARAH YOUNG| JANE SCHWEIGER|
596| GODS DREAM CATCHER| KAIYA KOLB, KAREN BANISTER| KAREN BANISTER|
598| MAGNOLIA| KAREN BANISTER| KAREN BANISTER|
606| HIGH TIME| ANNIE GRABER| ANNIE GRABER|
607| RINGO'S A STAR| RILEY GILBRETH| ALEXIA HONEGGER|
608| COURAGE| MEGAN SOUTHAM, EMERY VOLKERT| CINDY CRUCIOTTI|
609| NEIGH| GRACE HAMILTON| CINDY CRUCIOTTI|
610| INDEPENDENCE KF| ELIZABETH LATSIS, MEGAN SOUTHAM| CINDY CRUCIOTTI|
611| THREE RIVERS| JENNIFER WALLEN, KELLI CRUCIOTTI| CINDY CRUCIOTTI|
612| MARGO| MEGAN SOUTHAM, CASSANDRA RAGSDALE| CINDY CRUCIOTTI|
613| CEDRIC 42| HANNA GILHOOLEY| CINDY CRUCIOTTI|
614| MISS SCARLETT|GLORIA FOUNTAIN JAMES, MARIELLA MILLER| CINDY CRUCIOTTI|
615| ISHANA III| MEGAN SOUTHAM, JENNA KOLB| CINDY CRUCIOTTI|
616| REX ROMEO| ELIZABETH LATSIS, MEGAN SOUTHAM| CINDY CRUCIOTTI|
617| Q-T| KELLI CLEVENGER, CAMBELL BOWERS| CINDY CRUCIOTTI|
618| CAYMAN A.L.|ABIGAIL GRACE KELLEY, MEGAN SOUTHAM| CINDY CRUCIOTTI|
619| ALL INCLUSIVE| ABIGAIL GRACE KELLEY| CINDY CRUCIOTTI|
620| MARGARITA| EMILY ELEK-BURTARD| CINDY CRUCIOTTI|
621| SOUTHERN MARTINI| JOANNE JANSON| ALEXIA HONEGGER|
623| BALDO| KATIE GLASS| KRISTIN JACOB|