Question

我尝试访问ASPX网站，其中基于后续页面返回发布数据。不幸的是，我尝试获取以下页面失败了。希望有人知道在哪里可以找到错误！

在第一步中，我从cookie中读取会话ID以及该值返回的html页面中的viewstate变量。第二步打算发送它返回服务器以获取所需的页面。

嗅探webbrowser中的数据

Host=www.geocaching.com
User-Agent=Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100618
Iceweasel/3.5.9 (like Firefox/3.5.9)
Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language=en-us,en;q=0.5
Accept-Encoding=gzip,deflate
Accept-Charset=ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive=300
Connection=keep-alive
Referer=http://www.geocaching.com/seek/nearest.aspx?state_id=149
Cookie=Send2GPS=garmin; BMItemsPerPage=200; maprefreshlock=true; ASP.
NET_SessionId=c4jgygfvu1e4ft55dqjapj45
Content-Type=application/x-www-form-urlencoded
Content-Length=4099
POSTDATA=__EVENTTARGET=ctl00%24ContentBody%24pgrBottom%
24lbGoToPage_3&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPD[...]2Xg%3D%
3D&language=on&logcount=on&gpx=on

目前，我的脚本看起来像这样

import java.net.*;
import java.io.*;
import java.util.*;
import java.security.*;
import java.net.*;

public class test1 {
    public static void main(String args[]) {
        // String loginWebsite="http://geocaching.com/login/default.aspx";
        final String loginWebsite = "http://www.geocaching.com/seek/nearest.aspx?state_id=159";
        final String POST_CONTENT_TYPE = "application/x-www-form-urlencoded";

        // step 1: get session ID from cookie
        String sessionId = "";
        String viewstate = "";
        try {
            URL url = new URL(loginWebsite);

            String key = "";
            URLConnection urlConnection = url.openConnection();

            if (urlConnection != null) {
                for (int i = 1; (key = urlConnection.getHeaderFieldKey(i)) != null; i++) {
                    // get ASP.NET_SessionId from cookie
                    // System.out.println(urlConnection.getHeaderField(key));
                    if (key.equalsIgnoreCase("set-cookie")) {
                        sessionId = urlConnection.getHeaderField(key);
                        sessionId = sessionId.substring(0, sessionId.indexOf(";"));

                    }
                }

                BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));

                // get the viewstate parameter
                String aLine;
                while ((aLine = in.readLine()) != null) {
                    // System.out.println(aLine);
                    if (aLine.lastIndexOf("id=\"__VIEWSTATE\"") > 0) {
                        viewstate = aLine.substring(aLine.lastIndexOf("value=\"") + 7, aLine.lastIndexOf("\" "));
                    }
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

        System.out.println(sessionId);
        System.out.println("\n");
        System.out.println(viewstate);
        System.out.println("\n");

        // String goToPage="3";

        // step2: post data to site
        StringBuilder htmlResult = new StringBuilder();
        try {

            String encoded = "__EVENTTARGET=ctl00$ContentBody$pgrBottom$lbGoToPage_3" + "&" + "__EVENTARGUMENT=" + "&"
                + "__VIEWSTATE=" + viewstate;

            URL url = new URL(loginWebsite);
            URLConnection urlConnection = url.openConnection();
            urlConnection = url.openConnection();

            // Specifying that we intend to use this connection for input
            urlConnection.setDoInput(true);

            // Specifying that we intend to use this connection for output
            urlConnection.setDoOutput(true);

            // Specifying the content type of our post
            urlConnection.setRequestProperty("Content-Type", POST_CONTENT_TYPE);

            // urlConnection.setRequestMethod("POST");

            urlConnection.setRequestProperty("Cookie", sessionId);
            urlConnection.setRequestProperty("Content-Type", "text/html");

            DataOutputStream out = new DataOutputStream(urlConnection.getOutputStream());
            out.writeBytes(encoded);
            out.flush();
            out.close();

            BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));

            String aLine;
            while ((aLine = in.readLine()) != null) {
                System.out.println(aLine);
            }

        } catch (MalformedURLException e) {
            // Print out the exception that occurred
            System.err.println("Invalid URL " + e.getMessage());
        } catch (IOException e) {
            // Print out the exception that occurred
            System.err.println("Unable to execute " + e.getMessage());
        }
    }
}

知道什么是错的吗？非常感谢任何帮助！

更新

感谢您的快速回复！

我切换到使用HttpURLConnection而不是实现setRequestMethod（）的URLConnection。我还纠正了你提到的小错误，例如：删除了过时的第一个setRequestProperty调用。

不幸的是，这并没有改变任何东西......我想我设置了所有相关参数，但仍然只获得了列表的第一页。似乎忽略了“__EVENTTARGET = ctl00 $ ContentBody $ pgrBottom $ lbGoToPage_3”。我没有任何线索原因。

在网站上，网站上的表格如下：

<form name="aspnetForm" method="post" action="nearest.aspx?state_id=159" id="aspnetForm">

通过以下javascript调用：

<script type="text/javascript"> 
//<![CDATA[
var theForm = document.forms['aspnetForm'];
if (!theForm) {
    theForm = document.aspnetForm;
}
function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}
//]]>
</script>

希望这有助于找到解决方案吗？

问候 MAIK。

Answer 1

你真的想要GET或POST吗？如果你想POST，那么你可能需要setRequestMethd（）行。

您正在设置两次Content-Type - 我认为您可能需要将这些内容组合成一行。

然后，在尝试从输入流中读取之前，请不要关闭输出流。

除此之外，还有更多的日志记录，您可以提供/线索，您可以提供哪些方式出错？

Answer 2

嘿使用以下代码

String userAgent =“Mozilla / 5.0（Windows NT 6.1; Win64; x64; rv：25.0）Gecko / 20100101 Firefox / 25.0”;

        org.jsoup.nodes.Document jsoupDoc = Jsoup.connect(url).timeout(15000).userAgent(userAgent).referrer("http://calendar.legis.ga.gov/Calendar/?chamber=House").ignoreContentType(true)
                .data("__EVENTTARGET", eventtarget).data("__EVENTARGUMENT", eventarg).data("__VIEWSTATE", viewState).data("__VIEWSTATEGENERATOR", viewStateGenarator)
                .data("__EVENTVALIDATION", viewStateValidation).parser(Parser.xmlParser()).post();

使用POST数据和cookie处理网站

2 个答案: