我想抓取网站，我的代码中有一些问题，我希望所有数据都在列表中，但这给了我最后一页的数据。请帮助。 这是我的代码。...
import requests
import pandas as pd
from bs4 import BeautifulSoup
import numpy as np
url = 'https://www.imdb.com/list/ls025929404/?sort=list_order,asc&mode=detail&page='
for url_no in range(1,3,1):
r = requests.get(url+str(url_no))
print(url + str(url_no))
c = r.content
soup= BeautifulSoup(c,'html.parser') **// I am receiving here all data**
all = soup.find_all('div',{'class':'lister-list'}) **// but here i am receiving last page data**
for item in all:
images = item.find_all('div',{'class':'lister-item-image'})
name = item.find_all('h3',{'class':'lister-item-header'})
movie = item.find_all('p',{'class':'text-muted text-small'})
name_range=len(name)
movie_name = len(movie)
vvv = np.arange(1,len(all), 2)
vvv = list(vvv)
l= []
ll = []
i = 1
for names in range(name_range):
all_df= {}
all_df['Name']= name[names].text.replace('\n','').split('.')[1]
all_df['Movie'] = movie[names].text.replace('\n ','').replace('\n','').split(' | ')[1]
all_df['Profession'] = movie[names].text.replace('\n ','').replace('\n','').split(' | ')[0]
#l.append(all_df)
if i:
all_df['Info']= item.find_all('p')[i].text.replace('\n ','')
i+=2
l.append(all_df)
data = pd.DataFrame(l)
答案 0 :(得分：1)
在这里。使用空的DataFrame并附加数据。
import pandas as pd
url = 'https://www.imdb.com/list/ls025929404/?sort=list_order,asc&mode=detail&page={}'
df=pd.DataFrame()
for url_no in range(1,3,1):
r = requests.get(url.format(url_no))
print(url.format(url_no))
c = r.content
soup= BeautifulSoup(c,'html.parser')
all=soup.select(".lister-item.mode-detail ")
names=[item.select_one('.lister-item-image img[alt]')['alt'] for item in all]
movie=[item.select_one('.text-muted.text-small>a').text.strip() for item in all]
Profession=[item.select_one('.text-muted.text-small').next_element.strip() for item in all]
df1=pd.DataFrame({"Name":names,"Movie":movie,"Profession":Profession})
df=df.append(df1,ignore_index=True)
print(df)
输出：
https://www.imdb.com/list/ls025929404/?sort=list_order,asc&mode=detail&page=1
https://www.imdb.com/list/ls025929404/?sort=list_order,asc&mode=detail&page=2
Movie Name Profession
0 Gunga Jumna Dilip Kumar Actor
1 Mera Naam Joker Raj Kapoor Actor
2 Kala Bazar Dev Anand Actor
3 Black Amitabh Bachchan Actor
4 Sultan Salman Khan Actor
5 Anand Rajesh Khanna Actor
6 Don 2 Shah Rukh Khan Actor
7 Taare Zameen Par Aamir Khan Actor
8 Sholay Dharmendra Actor
9 Hera Pheri Akshay Kumar Actor
10 Mahal Ashok Kumar Actor
11 Mujhe Jeene Do Sunil Dutt Actor
12 Sholay Sanjeev Kumar Actor
13 Shivaay Ajay Devgn Actor
14 The League of Extraordinary Gentlemen Naseeruddin Shah Actor
15 Slumdog Millionaire Anil Kapoor Actor
16 Deewaar Shashi Kapoor Actor
17 Kalyug Ke Avtaar Jeetendra Actor
18 Munna Bhai M.B.B.S. Sanjay Dutt Actor
19 Arzoo Rajendra Kumar Actor
20 Qurbani Vinod Khanna Actor
21 Krantiveer Nana Patekar Actor
22 Vidhaata Shammi Kapoor Actor
23 Sahib Bibi Aur Ghulam Guru Dutt Actor
24 Ghayal Sunny Deol Actor
25 OMG: Oh My God! Mithun Chakraborty Actor
26 The Hundred-Foot Journey Om Puri Actor
27 Roti Kapada Aur Makaan Manoj Kumar Actor
28 Zanjeer Pran Actor
29 Mughal-E-Azam Prithviraj Kapoor Actor
.. ... ... ...
95 Main Hoon Na Zayed Khan Actor
96 Vicky Donor Annu Kapoor Actor
97 Andhadhun Ayushmann Khurrana Actor
98 Kapoor & Sons Fawad Khan Actor
99 Munna Bhai M.B.B.S. Arshad Warsi Actor
100 Masti Aftab Shivdasani Actor
101 Golmaal: Fun Unlimited Tusshar Kapoor Actor
102 Don Chunky Pandey Actor
103 3 Idiots Boman Irani Actor
104 Go Goa Gone Kunal Khemu Actor
105 Josh Chandrachur Singh Actor
106 A Wednesday Jimmy Sheirgill Actor
107 Jodhaa Akbar Sonu Sood Actor
108 Wanted Inder Kumar Actor
109 Johnny Gaddaar Neil Nitin Mukesh Actor
110 Aashiqui 2 Aditya Roy Kapoor Actor
111 Hero Sooraj Pancholi Actor
112 Everybody Says I'm Fine! Rahul Bose Actor
113 Sholay Asrani Actor
114 Bheja Fry Vinay Pathak Actor
115 Pyaar Kiya To Darna Kya Sohail Khan Producer
116 Rang De Basanti Kunal Kapoor Actor
117 3 Idiots Sharman Joshi Actor
118 3 Idiots Omi Vaidya Actor
119 Junoon Rahul Roy Actor
120 Qubool Hai Karan Singh Grover Actor
121 Udaan Ronit Roy Actor
122 Shahid Rajkummar Rao Actor
123 Prem Ratan Dhan Payo Arman Kohli Actor
124 Mirzya Harshvardhan Kapoor Actor
[125 rows x 3 columns]