如何使用bs4抓取网站?

时间:2020-04-27 12:18:32

标签: python-3.x web-scraping beautifulsoup web-crawler

我想抓取网站,我的代码中有一些问题,我希望所有数据都在列表中,但这给了我最后一页的数据。请帮助。 这是我的代码。...

import requests
import pandas as pd
from bs4 import BeautifulSoup
import numpy as np
url = 'https://www.imdb.com/list/ls025929404/?sort=list_order,asc&mode=detail&page='
for url_no in range(1,3,1):
    r = requests.get(url+str(url_no))
    print(url + str(url_no))
    c = r.content
    soup= BeautifulSoup(c,'html.parser') **// I am receiving here all data**
    all = soup.find_all('div',{'class':'lister-list'})  **// but here i am receiving last page data**
for item in all:
    images = item.find_all('div',{'class':'lister-item-image'})
    name = item.find_all('h3',{'class':'lister-item-header'})
    movie = item.find_all('p',{'class':'text-muted text-small'})
    name_range=len(name)
    movie_name = len(movie)
    vvv = np.arange(1,len(all), 2)
    vvv = list(vvv)
    l= []
    ll = []
    i = 1
    for names in range(name_range):
        all_df= {}
        all_df['Name']= name[names].text.replace('\n','').split('.')[1]
        all_df['Movie'] = movie[names].text.replace('\n                        ','').replace('\n','').split(' | ')[1]
        all_df['Profession'] = movie[names].text.replace('\n                        ','').replace('\n','').split(' | ')[0]
        #l.append(all_df)

        if i:
            all_df['Info']= item.find_all('p')[i].text.replace('\n ','')
            i+=2
            l.append(all_df)

data = pd.DataFrame(l)

1 个答案:

答案 0 :(得分:1)

在这里。使用空的DataFrame并附加数据。

import pandas as pd

url = 'https://www.imdb.com/list/ls025929404/?sort=list_order,asc&mode=detail&page={}'
df=pd.DataFrame()
for url_no in range(1,3,1):
    r = requests.get(url.format(url_no))
    print(url.format(url_no))
    c = r.content
    soup= BeautifulSoup(c,'html.parser')
    all=soup.select(".lister-item.mode-detail ")
    names=[item.select_one('.lister-item-image img[alt]')['alt'] for item in all]
    movie=[item.select_one('.text-muted.text-small>a').text.strip() for item in all]
    Profession=[item.select_one('.text-muted.text-small').next_element.strip() for item in all]

    df1=pd.DataFrame({"Name":names,"Movie":movie,"Profession":Profession})
    df=df.append(df1,ignore_index=True)


print(df)

输出

https://www.imdb.com/list/ls025929404/?sort=list_order,asc&mode=detail&page=1
https://www.imdb.com/list/ls025929404/?sort=list_order,asc&mode=detail&page=2
                                     Movie                 Name Profession
0                              Gunga Jumna          Dilip Kumar      Actor
1                          Mera Naam Joker           Raj Kapoor      Actor
2                               Kala Bazar            Dev Anand      Actor
3                                    Black     Amitabh Bachchan      Actor
4                                   Sultan          Salman Khan      Actor
5                                    Anand        Rajesh Khanna      Actor
6                                    Don 2       Shah Rukh Khan      Actor
7                         Taare Zameen Par           Aamir Khan      Actor
8                                   Sholay           Dharmendra      Actor
9                               Hera Pheri         Akshay Kumar      Actor
10                                   Mahal          Ashok Kumar      Actor
11                          Mujhe Jeene Do           Sunil Dutt      Actor
12                                  Sholay        Sanjeev Kumar      Actor
13                                 Shivaay           Ajay Devgn      Actor
14   The League of Extraordinary Gentlemen     Naseeruddin Shah      Actor
15                     Slumdog Millionaire          Anil Kapoor      Actor
16                                 Deewaar        Shashi Kapoor      Actor
17                        Kalyug Ke Avtaar            Jeetendra      Actor
18                     Munna Bhai M.B.B.S.          Sanjay Dutt      Actor
19                                   Arzoo       Rajendra Kumar      Actor
20                                 Qurbani         Vinod Khanna      Actor
21                              Krantiveer         Nana Patekar      Actor
22                                Vidhaata        Shammi Kapoor      Actor
23                   Sahib Bibi Aur Ghulam            Guru Dutt      Actor
24                                  Ghayal           Sunny Deol      Actor
25                         OMG: Oh My God!   Mithun Chakraborty      Actor
26                The Hundred-Foot Journey              Om Puri      Actor
27                  Roti Kapada Aur Makaan          Manoj Kumar      Actor
28                                 Zanjeer                 Pran      Actor
29                           Mughal-E-Azam    Prithviraj Kapoor      Actor
..                                     ...                  ...        ...
95                            Main Hoon Na           Zayed Khan      Actor
96                             Vicky Donor          Annu Kapoor      Actor
97                               Andhadhun   Ayushmann Khurrana      Actor
98                           Kapoor & Sons           Fawad Khan      Actor
99                     Munna Bhai M.B.B.S.         Arshad Warsi      Actor
100                                  Masti     Aftab Shivdasani      Actor
101                 Golmaal: Fun Unlimited       Tusshar Kapoor      Actor
102                                    Don        Chunky Pandey      Actor
103                               3 Idiots          Boman Irani      Actor
104                            Go Goa Gone          Kunal Khemu      Actor
105                                   Josh    Chandrachur Singh      Actor
106                            A Wednesday      Jimmy Sheirgill      Actor
107                           Jodhaa Akbar            Sonu Sood      Actor
108                                 Wanted          Inder Kumar      Actor
109                         Johnny Gaddaar    Neil Nitin Mukesh      Actor
110                             Aashiqui 2    Aditya Roy Kapoor      Actor
111                                   Hero      Sooraj Pancholi      Actor
112               Everybody Says I'm Fine!           Rahul Bose      Actor
113                                 Sholay               Asrani      Actor
114                              Bheja Fry         Vinay Pathak      Actor
115                Pyaar Kiya To Darna Kya          Sohail Khan   Producer
116                        Rang De Basanti         Kunal Kapoor      Actor
117                               3 Idiots        Sharman Joshi      Actor
118                               3 Idiots           Omi Vaidya      Actor
119                                 Junoon            Rahul Roy      Actor
120                             Qubool Hai   Karan Singh Grover      Actor
121                                  Udaan            Ronit Roy      Actor
122                                 Shahid        Rajkummar Rao      Actor
123                   Prem Ratan Dhan Payo          Arman Kohli      Actor
124                                 Mirzya  Harshvardhan Kapoor      Actor

[125 rows x 3 columns]