Question

我正在尝试使用beautifulsoup学习网络抓取，并且已经实现了此代码。但是，虽然已经检索了两个电影标题，但只有电影标题被写入了csv文件，而Genre则没有。

URL：http://www.imdb.com/search/title?sort=num_votes,desc&start=1&title_type=feature&year=1950,2012

f = csv.writer(open('movie-names.csv', 'w'))
f.writerow(['Title', 'Genre'])

pages = []
genre;


for i in range(1,2):
    url = 'http://www.imdb.com/search/title?sort=num_votes,desc&start=1&title_type=feature&year=1950,2012'
    pages.append(url)


for item in pages:
    page = requests.get(item)
    soup = BeautifulSoup(page.text, 'html.parser')

    movie_titles = soup.find_all(class_ = 'lister-item-content')

    for movie_title in movie_titles:
        title = movie_title.find('a').contents[0]
        genre = movie_title.find_all(class_ = 'genre')[0].get_text()
        print(genre)
        f.writerow([title, genre])

Answer 1

使用pandas可以更轻松地导出CSV中的数据。

from bs4 import BeautifulSoup
import requests
import pandas as pd
pages = []

for i in range(1,2):
    url = 'http://www.imdb.com/search/title?sort=num_votes,desc&start=1&title_type=feature&year=1950,2012'
    pages.append(url)

Movie_title=[]
Movie_genre=[]
for item in pages:
    page = requests.get(item)
    soup = BeautifulSoup(page.text, 'html.parser')

    movie_titles = soup.select('.lister-item-content')

    for movie_title in movie_titles:
        title = movie_title.select_one('a').text
        Movie_title.append(title)
        genre = movie_title.select_one('.genre').text.replace('\n','')
        Movie_genre.append(genre)


df = pd.DataFrame({"Movie_title":Movie_title,"Movie_genre":Movie_genre})
df.to_csv("movie-names.csv",index=False)

输出：

Answer 2

这应该有效：

import requests
from bs4 import BeautifulSoup
import csv

with open("movie-names.csv", "w") as f:
    writer = csv.writer(f)
    writer.writerow(['Title', 'Genre'])

    pages = []
    genre = []


    for i in range(1,2):
        url = 'http://www.imdb.com/search/title?sort=num_votes,desc&start=1&title_type=feature&year=1950,2012'
        pages.append(url)


    for item in pages:
        page = requests.get(item)
        soup = BeautifulSoup(page.text, 'html.parser')

        movie_titles = soup.find_all(class_ = 'lister-item-content')

        for movie_title in movie_titles:
            title = movie_title.find('a').contents[0]
            genre = movie_title.find_all(class_ = 'genre')[0].get_text()
            print(title, genre)
            writer.writerow([title, genre])

这是我运行代码的.csv中内容的摘录：

Title   Genre
The Shawshank Redemption     Drama            
The Dark Knight  Action, Crime, Drama            
Inception    Action, Adventure, Sci-Fi            
Fight Club   Drama            
Pulp Fiction     Crime, Drama            
Forrest Gump     Drama, Romance

请注意，此for循环：

for i in range(1,2):
    url = 'http://www.imdb.com/search/title?sort=num_votes,desc&start=1&title_type=feature&year=1950,2012'
    pages.append(url)

将毫无用处，因为在这种情况下，仅添加一个URL。在2为n的一般情况下，它会附加相同个网址n-1次。那是你的意图吗？

为什么在我的代码中“流派”数据未写入.csv文件

2 个答案: