我试图抓取网站的所有结果页面,但它有效,但有时脚本停止并显示此错误:
502 => Net::HTTPBadGateway for https://website.com/id/12/ -- unhandled response (Mechanize::ResponseCodeError)
即使发现错误,我也希望继续使用该脚本。
我的剧本:
require 'mechanize'
require 'csv'
a = Mechanize.new
CSV.open('datas.csv', "wb") do |csv|
page = a.get("https://website.com/?page=1-200") #498
number = 0
page.links_with(:class => "btn btn-default").each do |link|
post_link = link.href
inside_page = a.get("https://website.com#{post_link}")
title = inside_page.at("h1.serviceTitle").text.strip
author = inside_page.at(".name").text.strip
number+=1
csv << [title, author]
end
end
有什么想法吗?
答案 0 :(得分:0)
这可以通过适当的异常处理轻松解决。你可以check this page for a better explanation
至于你的代码,你可以像这样处理异常
CSV.open('datas.csv', "wb") do |csv|
begin
a = Mechanize.new
page = a.get("https://website.com/?page=1-200") #498
number = 0
page.links_with(:class => "btn btn-default").each do |link|
post_link = link.href
inside_page = a.get("https://website.com#{post_link}")
title = inside_page.at("h1.serviceTitle").text.strip
author = inside_page.at(".name").text.strip
number+=1
csv << [title, author]
end
rescue => e
// do nothing and move on to the next line
end
end