Question

我正在尝试抓取this webpage，因为我们滚动它加载时它有延迟加载。使用Nokogiri我能够抓取初始页面，但不能滚动页面的其余部分。

Answer 1

要获取延迟加载的页面，请废弃以下页面：

http://www.flipkart.com/mens-footwear/shoes/casual-shoes/pr?p%5B%5D=sort%3Dpopularity&sid=osp%2Ccil%2Cnit%2Ce1f&start=31&ajax=true
http://www.flipkart.com/mens-footwear/shoes/casual-shoes/pr?p%5B%5D=sort%3Dpopularity&sid=osp%2Ccil%2Cnit%2Ce1f&start=46&ajax=true
http://www.flipkart.com/mens-footwear/shoes/casual-shoes/pr?p%5B%5D=sort%3Dpopularity&sid=osp%2Ccil%2Cnit%2Ce1f&start=61&ajax=true
...

require 'rubygems'
require 'nokogiri'
require 'mechanize'
require 'open-uri'

number = 1
while true
  url = "http://www.flipkart.com/mens-footwear/shoes" +
        "/casual-shoes/pr?p%5B%5D=sort%3Dpopularity&" +
        "sid=osp%2Ccil%2Cnit%2Ce1f&start=#{number}&ajax=true"

  doc = Nokogiri::HTML(open(url))
  doc = Nokogiri::HTML(doc.at_css('#ajax').text)

  products = doc.css(".browse-product")
  break if products.size == 0

  products.each do |item|
    title = item.at_css(".fk-display-block,.title").text.strip
    price = (item.at_css(".pu-final").text || '').strip
    link = item.at_xpath(".//a[@class='fk-display-block']/@href")
    image = item.at_xpath(".//div/a/img/@src")

    puts number
    puts "#{title} - #{price}"
    puts "http://www.flipkart.com#{link}"
    puts image
    puts "========================"

    number += 1
  end
end

如何使用JavaScript添加动态内容来抓取网页？

1 个答案: