使用JSOUP在.aspx网站上使用POST请求进行抓取时无法获取新页面

时间:2018-04-11 11:07:14

标签: asp.net http web-scraping jsoup htmlunit

我想从特定网站抓取数据。它有一些过滤器需要选择来获取数据。我正在尝试使用JSOUP并发布表单数据并在过滤器中获取新选择的html。以下是我的代码。每次我尝试运行程序时,我只得到默认页面html,即,没有获得具有新过滤器输入数据的页面。

        Document doc = Jsoup.connect("http://45.114.246.184/bsmm/Dashboards/DB.aspx").timeout(50*1000).get();

    Elements viewState = doc.select("input[name=__VIEWSTATE");
    Elements viewStateGen = doc.select("input[name=__VIEWSTATEGENERATOR");
    Elements eventValidation = doc.select("input[name=__EVENTVALIDATION]");

    Map<String,String> allFields = new HashMap<String,String>();
    allFields.put("ctl00$ScriptManager1", "ctl00$ContentPlaceHolder1$updatepanel|ctl00$ContentPlaceHolder1$btnfilter");
    allFields.put("ctl00$ContentPlaceHolder1$ddltop", "1");
    allFields.put("ctl00$ContentPlaceHolder1$ddl1", "12");
    allFields.put("ctl00$ContentPlaceHolder1$ddl2", "117");
    allFields.put("__EVENTTARGET", "");
    allFields.put("__EVENTARGUMENT", "");
    allFields.put("__LASTFOCUS", "");
    allFields.put("__VIEWSTATE", viewState.val());
    allFields.put("__EVENTVALIDATION", eventValidation.val());
    allFields.put("__VIEWSTATEGENERATOR", viewStateGen.val());
    allFields.put("__ASYNCPOST", "true");
    allFields.put("ctl00$ContentPlaceHolder1$btnfilter", "Submit");
    allFields.put("ctl00$ContentPlaceHolder1$btnrefresh", "Reset");


    Connection.Response res = Jsoup.connect("http://45.114.246.184/bsmm/Dashboards/DB.aspx")
            //.userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36")
            .data(allFields)
            .timeout(50*1000)
            .method(Method.POST).
            execute();

    Document doc2  = Jsoup.connect("http://45.114.246.184/bsmm/Dashboards/DB.aspx")
            .cookies(res.cookies())
            .timeout(50*1000)
            .get();

    System.out.println(doc2);

我做错了什么只是JSOUP在这种情况下不是正确的工具?

提前致谢。

0 个答案:

没有答案