我要从以下页面抓取电影标题：https://www.imdb.com/list/ls055386972/。我写了以下代码：
import scrapy
from scrapy import Spider
from scrapy.http import Request
import re
import pymysql
import sys
import hashlib
from datetime import *
#import time
import csv
import os
import requests
class MoviesSpider(scrapy.Spider):
name = 'movies' #name of the spider
allowed_domains = ['imdb.com/list/ls055386972/']
start_urls = ['http://imdb.com/list/ls055386972//']
def parse(self, response):
#events = response.xpath('//*[@property="url"]/@href').extract()
links = response.xpath('//h3[@class]/a/@href').extract()
final_links = []
for link in links:
final_link = 'http://www.imdb.com' + link
final_links.append(final_link)
for final_link in final_links:
absolute_url = response.urljoin(final_link)
yield Request(absolute_url, callback = self.parse_movies)
#process next page url
#next_page_url = response.xpath('//a[text() = "Next"]/@href').extract_first()
#absolute_next_page_url = response.urljoin(next_page_url)
#yield Request(absolute_next_page_url)
def parse_movies(self, response):
title = response.xpath('//div[@class = "title_wrapper"]/h1[@class]/text()').extract_first()
yield{
'title': title,
}
但是它没有刮擦任何东西。我收到此错误消息：
2019-03-04 18:08:37 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://www.imdb.com/list/ls055386972//> (referer: None)
2019-03-04 18:08:37 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.imdb.com/list/ls055386972//>: HTTP status code is not handled or not allowed
2019-03-04 18:08:37 [scrapy.core.engine] INFO: Closing spider (finished)
print（'final_links'）产生指向各个电影页面的正确链接：
[u'https://www.imdb.com/title/tt0068646/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0108052/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0050083/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0118799/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0060196/?ref_=ttls_li_tt',..........]
答案 0 :(得分：2)
您收到404，因为您的起始网址不正确。您需要删除start_urls中的尾部正斜杠：
start_urls = ['https://www.imdb.com/list/ls055386972/']
此外，您的allowed_domains不正确。它应仅包含域，而不包含部分URL：
allowed_domains = ['imdb.com']
请参阅documentation。
答案 1 :(得分：0)
不确定Scrapy，但是如果使用下面的代码，则会得到所需的输出。
import requests
import lxml
from lxml import html
url = "https://www.imdb.com/list/ls055386972/"
r = requests.get(url)
tree = html.fromstring(r.content)
domain = "https://www.imdb.com/"
movies_list = tree.xpath('//div[@class="lister-item-content"]/h3/a')
for i, movie in enumerate(movies_list, 1):
print(i, movie.text, domain+movie.get('href'))
1 The Godfather https://www.imdb.com//title/tt0068646/?ref_=ttls_li_tt
2 Schindler's List https://www.imdb.com//title/tt0108052/?ref_=ttls_li_tt
3 12 Angry Men https://www.imdb.com//title/tt0050083/?ref_=ttls_li_tt
4 Life Is Beautiful https://www.imdb.com//title/tt0118799/?ref_=ttls_li_tt
5 The Good, the Bad and the Ugly https://www.imdb.com//title/tt0060196/?ref_=ttls_li_tt
6 The Shawshank Redemption https://www.imdb.com//title/tt0111161/?ref_=ttls_li_tt
7 The Pursuit of Happyness https://www.imdb.com//title/tt0454921/?ref_=ttls_li_tt
8 Seven Samurai https://www.imdb.com//title/tt0047478/?ref_=ttls_li_tt
9 The Intouchables https://www.imdb.com//title/tt1675434/?ref_=ttls_li_tt
10 Central Station https://www.imdb.com//title/tt0140888/?ref_=ttls_li_tt
11 Requiem for a Dream https://www.imdb.com//title/tt0180093/?ref_=ttls_li_tt
12 A Beautiful Mind https://www.imdb.com//title/tt0268978/?ref_=ttls_li_tt
13 Hachi: A Dog's Tale https://www.imdb.com//title/tt1028532/?ref_=ttls_li_tt
14 Taken https://www.imdb.com//title/tt0936501/?ref_=ttls_li_tt
15 My Sassy Girl https://www.imdb.com//title/tt0293715/?ref_=ttls_li_tt
16 Amores Perros https://www.imdb.com//title/tt0245712/?ref_=ttls_li_tt
17 The Shining https://www.imdb.com//title/tt0081505/?ref_=ttls_li_tt
18 Apocalypto https://www.imdb.com//title/tt0472043/?ref_=ttls_li_tt
19 Gladiator https://www.imdb.com//title/tt0172495/?ref_=ttls_li_tt
20 Cast Away https://www.imdb.com//title/tt0162222/?ref_=ttls_li_tt
21 The Dark Knight https://www.imdb.com//title/tt0468569/?ref_=ttls_li_tt
22 The Pianist https://www.imdb.com//title/tt0253474/?ref_=ttls_li_tt
23 Titanic https://www.imdb.com//title/tt0120338/?ref_=ttls_li_tt
24 3-Iron https://www.imdb.com//title/tt0423866/?ref_=ttls_li_tt
25 Braveheart https://www.imdb.com//title/tt0112573/?ref_=ttls_li_tt
26 It's a Wonderful Life https://www.imdb.com//title/tt0038650/?ref_=ttls_li_tt
27 Spring, Summer, Fall, Winter... and Spring https://www.imdb.com//title/tt0374546/?ref_=ttls_li_tt
28 Alien https://www.imdb.com//title/tt0078748/?ref_=ttls_li_tt
29 Memories of Murder https://www.imdb.com//title/tt0353969/?ref_=ttls_li_tt
30 The Return https://www.imdb.com//title/tt0376968/?ref_=ttls_li_tt
31 I Saw the Devil https://www.imdb.com//title/tt1588170/?ref_=ttls_li_tt
32 Children of Heaven https://www.imdb.com//title/tt0118849/?ref_=ttls_li_tt
33 A Separation https://www.imdb.com//title/tt1832382/?ref_=ttls_li_tt
34 The Sixth Sense https://www.imdb.com//title/tt0167404/?ref_=ttls_li_tt
35 A Moment to Remember https://www.imdb.com//title/tt0428870/?ref_=ttls_li_tt
36 Departures https://www.imdb.com//title/tt1069238/?ref_=ttls_li_tt
37 Wo de fu qin mu qin https://www.imdb.com//title/tt0235060/?ref_=ttls_li_tt
38 Saving Private Ryan https://www.imdb.com//title/tt0120815/?ref_=ttls_li_tt
39 The Bridge on the River Kwai https://www.imdb.com//title/tt0050212/?ref_=ttls_li_tt
40 Ben-Hur https://www.imdb.com//title/tt0052618/?ref_=ttls_li_tt
41 The Exorcist https://www.imdb.com//title/tt0070047/?ref_=ttls_li_tt
42 The Secret in Their Eyes https://www.imdb.com//title/tt1305806/?ref_=ttls_li_tt
43 Léon: The Professional https://www.imdb.com//title/tt0110413/?ref_=ttls_li_tt
44 The Green Mile https://www.imdb.com//title/tt0120689/?ref_=ttls_li_tt
45 Gran Torino https://www.imdb.com//title/tt1205489/?ref_=ttls_li_tt
46 Kill Bill: Vol. 1 https://www.imdb.com//title/tt0266697/?ref_=ttls_li_tt
47 Jurassic Park https://www.imdb.com//title/tt0107290/?ref_=ttls_li_tt
48 Terminator 2: Judgment Day https://www.imdb.com//title/tt0103064/?ref_=ttls_li_tt
49 Back to the Future https://www.imdb.com//title/tt0088763/?ref_=ttls_li_tt
50 Finding Nemo https://www.imdb.com//title/tt0266543/?ref_=ttls_li_tt
答案 2 :(得分：0)
您尚未将起始网址解析为解析功能，以下为工作代码。
echo PATH="/usr/local/lib/ruby/gems/2.5.0/bin:$PATH" >> ~/.zshrc