我正在尝试使用Python和Beautiful Soup从特定页面中提取一些信息。我发现即使我请求不同的网页，request.get(url)也无法更改页面。
在下面一行中，我定义了我正在抓取的页面，所有页面都返回pagenumber=1，即使我尝试使用pagenumber=2，它也会从第一页开始，只抓取第一页。
activepage = soup.find('ul', id= 'pagination').li.string
print "Page Number: " + activepage
我在其他页面上测试了我的代码并且工作正常，但在这个特定的页面上，我无法遍历不同的页面。任何人都可以告诉我这个页面的确切问题是什么，解决方案是什么？
import requests
import sys
from bs4 import BeautifulSoup
def trade_spider(max_pages):
page_number = 1
while page_number <= max_pages:
url = "http://munich.eventful.com/events/categories/festivals_parades#!page_number=" + str(page_number) + "&category=festivals_parades"
source_code = requests.get(url)
# just get the code, no headers or anything
plain_text = source_code.text
# BeautifulSoup objects can be sorted through easy
soup = BeautifulSoup(plain_text)
category = soup.find('li', id = 'breadcrumb-label').string
activepage = soup.find('ul', id= 'pagination').li.string
print "Page Number: " + activepage
for mylist in soup.findAll('li', {'class': 'clearfix'}):
link = mylist.find('a', {'data-ga-label': 'Event Title'})
if (link is not None):
href = link.get('href')
title = link.string # just the text, not the HTML
location = mylist.find("div", {"class": "event-meta"}).strong.string
date = mylist.find("div", {"class": "event-meta"}).span.string
print(title, category, href, date, location)
page_number += 1
trade_spider(8)
答案 0 :(得分：0)
这应该得到你想要的东西：
def trade_spider(max_pages):
for page_number in range(1, max_pages + 1):
url = "http://munich.eventful.com/events/categories/festivals_parades?page_number={}".format(page_number)
print(url)
# just get the code, no headers or anything
plain_text = requests.get(url).content
# BeautifulSoup objects can be sorted through easy
soup = BeautifulSoup(plain_text)
category = soup.find('li', id='breadcrumb-label').string
for mylist in soup.findAll('li', {'class': 'clearfix'}):
link = mylist.find('a', {'data-ga-label': 'Event Title'})
if link is not None:
href = link.get('href')
title = link.string # just the text, not the HTML
location = mylist.find("div", {"class": "event-meta"}).strong.string
date = mylist.find("div", {"class": "event-meta"}).span.string
print(title, category, href, date, location)
我只需要更改基本网址：
http://munich.eventful.com/events/categories/festivals_parades?page_number={}
在range(1, max_pages + 1)
输出：
trade_spider(3)
http://munich.eventful.com/events/categories/festivals_parades?page_number=1
Rockavaria 2015 - Freitag Festivals http://munich.eventful.com/events/rockavaria-2015-freitag-/E0-001-077521284-1 Kleine Olympiahalle May 29
Rockavaria 2015 - Reiseangebote Festivals http://munich.eventful.com/events/rockavaria-2015-reiseangebote-/E0-001-077133997-3 Kleine Olympiahalle May 29 - May 31
Rockavaria 2015 - Samstag Festivals http://munich.eventful.com/events/rockavaria-2015-samstag-/E0-001-082047581-8 Kleine Olympiahalle May 30
Ermittelt wird im Cafe`Mellow München... Festivals http://munich.eventful.com/events/ermittelt-wird-im-cafemellow-mnchen-tatort-im-ma-/E0-001-083732894-3 Cafe Bar Mellow München May 24
Ermittelt wird im Cafe`Mellow München... Festivals http://munich.eventful.com/events/ermittelt-wird-im-cafemellow-mnchen-tatort-im-ma-/E0-001-083934156-8 Cafe Bar Mellow München May 31
Ermittelt wird im Cafe`Mellow München... Festivals http://munich.eventful.com/events/ermittelt-wird-im-cafemellow-mnchen-tatort-im-ma-/E0-001-084050435-8 Cafe Bar Mellow München May 25
ORGANIC DANCE MUSIC FESTIVAL Festivals http://munich.eventful.com/events/organic-dance-music-festival-/E0-001-076992422-8 Zenith Jun 13
Organic Dance Music Festival with Tal... Festivals http://munich.eventful.com/events/organic-dance-music-festival-tale-us-dixon-david-/E0-001-082231599-0 Zenith Jun 13
13. aDevantgarde-Festival »humus« Wur... Festivals http://munich.eventful.com/events/13-adevantgardefestival-humus-wurzel3-doppelkonz-/E0-001-082515660-8 Gasteig Jun 14
http://munich.eventful.com/events/categories/festivals_parades?page_number=2
Festive Concerts: Palace Schleißheim Festivals http://munich.eventful.com/events/festive-concerts-palace-schleiheim-/E0-001-082934543-1 Schloss Schleissheim Jul 5
Festive Concerts: Palace Schleißheim Festivals http://munich.eventful.com/events/festive-concerts-palace-schleiheim-/E0-001-082934542-2 Schloss Schleissheim Jun 19
Festive Concerts: Palace Schleißheim Festivals http://munich.eventful.com/events/festive-concerts-palace-schleiheim-/E0-001-082957294-7 Schloss Schleissheim Aug 2
Festive Concerts: Palace Schleißheim Festivals http://munich.eventful.com/events/festive-concerts-palace-schleiheim-/E0-001-082957292-9 Schloss Schleissheim Jun 14
Tollwood Festival: Patti Smith Festivals http://munich.eventful.com/events/tollwood-festival-patti-smith-/E0-001-080907608-3 Tollwood Jul 13
VLAD IN TEARS - FREE AND EASY FESTIVAL Festivals http://munich.eventful.com/events/vlad-tears-free-and-easy-festival-/E0-001-083500016-8 Backstage Jul 25
COMBICHRIST - FREE AND EASY FESTIVAL Festivals http://munich.eventful.com/events/combichrist-free-and-easy-festival-/E0-001-083499106-7 Backstage Jul 25
13. aDevantgarde-Festival »humus« Spi... Festivals http://munich.eventful.com/events/13-adevantgardefestival-humus-spinnen-/E0-001-082515659-2 Gasteig Jun 15
Barber & Cook - EOS Festival Festivals http://munich.eventful.com/events/barber-cook-eos-festival-/E0-001-081892354-3 Taufkirchen (Vils) Jun 4
http://munich.eventful.com/events/categories/festivals_parades?page_number=3
Yaz Aski Indoor Festival - Mabel Mati... Festivals http://eventful.com/events/yaz-aski-indoor-festival-mabel-matiz-ilyas-yalcintas-karga-/E0-001-083819488-4 Muffathalle Jun 6
Keep It Low Festival 2015 MÜNCHEN Festivals http://munich.eventful.com/events/keep-low-festival-2015-mnchen-/E0-001-083819658-1 Feierwerk Oct 16
10. Oberhachinger Classic Jazz Festiv... Festivals http://munich.eventful.com/events/10-oberhachinger-classic-jazz-festival-3-tag-/E0-001-080921159-8 Bürgersaal Beim Forstner Jun 13
10. Oberhachinger Classic Jazz Festiv... Festivals http://munich.eventful.com/events/10-oberhachinger-classic-jazz-festival-2-tag-/E0-001-080921158-9 Bürgersaal Beim Forstner Jun 12
Kool & Kabul - EOS Festival Festivals http://munich.eventful.com/events/kool-kabul-eos-festival-/E0-001-083947944-5 Crazy Town Jun 4
10. Oberhachinger Classic Jazz Festiv... Festivals http://munich.eventful.com/events/10-oberhachinger-classic-jazz-festival-1-tag-e-/E0-001-080921157-0 Bürgersaal Beim Forstner Jun 11
Mellow Monks - Psychedelic Happiness ... Festivals http://munich.eventful.com/events/mellow-monks-psychedelic-happiness-festival-201-/E0-001-078583630-3 Munich, Bayern, Germany Jun 12
Organic Dance Music Festival MÜNCHEN ... Festivals http://munich.eventful.com/events/organic-dance-music-festival-mnchen-freimann-/E0-001-081520443-8 Zenith & Kesselhaus + dazwischenliegende Freifl... Jun 13
aDevantgarde Festival: Young Lions Re... Festivals http://munich.eventful.com/events/adevantgarde-festival-young-lions-reloaded-/E0-001-083294075-9 Club Milla Jun 13