Question

我正在尝试从网页的99页中抓取某些元素。我一辈子都无法弄清楚该怎么做。这是我的代码：

require 'open-uri'
require 'nokogiri'
@title = []
html_content = open("https://www.imdb.com/list/ls057823854/? 
sort=list_order,asc&st_dt=&mode=detail&page=1").read
doc = Nokogiri::HTML(html_content)
doc.search(".lister-item-header/a").each do |title|
@title << title.text.strip

Answer 1

如果您想收集所有标题，请输入剪贴簿代码。

require 'open-uri'
require 'nokogiri'
require 'json'

@title = []
url = "https://www.imdb.com/list/ls057823854/?sort=list_order,asc&st_dt=&mode=detail&page="
html_content = open(url+"1").read
doc = Nokogiri::HTML(html_content)

max = doc.search(".pagination-range").first.text.split("of")[1].gsub(",","").strip.to_i
max = (max / 100).floor + 1

doc.search(".lister-item-header/a").each do |title|
    @title << title.text.strip
end

for i in 2..max
    html_content = open(url+i.to_s).read
    doc = Nokogiri::HTML(html_content)

    doc.search(".lister-item-header/a").each do |title|
        @title << title.text.strip
    end
    sleep(1) 
end

File.open("imdb-titles.json","w") do |f|
    f.write(JSON.pretty_generate(@title))
end

如何使用nokogiri抓取多个页面以及如何使用rails快速抓取

1 个答案: