Question

import com.jaunt.*;
public class JauntCrawler{
  public static void main(String[] args){
    try{
        UserAgent userAgent = new UserAgent();         //create new userAgent (headless browser)
        userAgent.visit("http://google.de");          //visit google
        userAgent.doc.apply("schmetterlinge");            //apply form input (starting at first editable field)
        userAgent.doc.submit();         //click submit button labelled "Google Search"


        Elements links = userAgent.doc.findEvery("<h3 class=r>").findEvery("<a>");  //find search result links
        for(Element link : links) System.out.println(link.getAt("href"));           //print results

        if(userAgent.doc.nextPageLinkExists()) {
            userAgent.visit(userAgent.doc.nextPageLink().getHref());
            Elements newlinks = userAgent.doc.findEvery("<h3 class=r>").findEvery("<a>");
            System.out.println("\nPage 2:");
            for(Element link : newlinks) System.out.println(link.getAt("href"));
        }
    }
    catch(JauntException e){         //if an HTTP/connection error occurs, handle JauntException.
      System.err.println(e);
    }
  }
}

我想从Google返回的搜索结果多于第一页。所以第二个for-loop应该基本上返回下一页的结果，但事实并非如此。知道为什么吗？

Answer 1

我也遇到了同样的问题。用户代理不会进入下一页，但我找到了另一种方法来实现这一目标：

Elements nextLinks = userAgent.doc.findEvery("<a class=fl");
        for(int i=0;i<nextLinks.size();i++) {
            userAgent.visit("http://google.co.in/search?q="+<search_string+"&start="+(i+1)*10);
            links = userAgent.doc.findEvery("<h3 class=r>").findEvery("<a>"); 
            for(Element link : links) System.out.println(link.getAt("href"));
        }

Jaunt webcrawler - 无法访问Google搜索结果的下一页

1 个答案: