Question

我在Java中使用HTMLUnit连接到远程URL并从获得的网页中获取一些信息。

我使用以下代码：

final WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_6_0, "companyproxy.server", 8080);
final DefaultCredentialsProvider scp = new DefaultCredentialsProvider();
scp.addProxyCredentials("username", "password","companyproxy.server",8080);
webClient.setCredentialsProvider(scp);

final URL url = new URL("http://htmlunit.sourceforge.net");
final HtmlPage page = (HtmlPage)webClient.getPage(url);
System.out.println(page.asXml());

在提供代理服务器的详细信息后，我收到此错误消息：

SEVERE: Credentials cannot be used for NTLM authentication:
org.apache.commons.httpclient.UsernamePasswordCredentials
org.apache.commons.httpclient.auth.InvalidCredentialsException: Credentials cannot be used for NTLM authentication: org.apache.commons.httpclient.UsernamePasswordCredentials
    at org.apache.commons.httpclient.auth.NTLMScheme.authenticate(NTLMScheme.java:332)
    at org.apache.commons.httpclient.HttpMethodDirector.authenticateProxy(HttpMethodDirector.java:320)
    at org.apache.commons.httpclient.HttpMethodDirector.authenticate(HttpMethodDirector.java:232)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:97)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1477)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1435)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:327)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:388)
    at com.test.Test.main(Test.java:25)
Jun 5, 2009 9:28:35 AM org.apache.commons.httpclient.HttpMethodDirector processProxyAuthChallenge
INFO: Failure authenticating with NTLM <any realm>@companyproxy.server:8080
Jun 5, 2009 9:28:35 AM com.gargoylesoftware.htmlunit.WebClient printContentIfNecessary
INFO: statusCode=[407] contentType=[text/html]
Jun 5, 2009 9:28:35 AM com.gargoylesoftware.htmlunit.WebClient printContentIfNecessary
INFO: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>

...

Exception in thread "main" com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException:

407 Proxy Authentication Required for http://htmlunit.sourceforge.net/
    at com.gargoylesoftware.htmlunit.WebClient.throwFailingHttpStatusCodeExceptionIfNecessary(WebClient.java:535)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:332)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:388)
    at com.test.Test.main(Test.java:25)

您能否提供一些相关信息？

Answer 1

我遇到了同样的问题，并在网上找到了解决方案。忘记 setCredentialsProvider（）。使用此：

String userAndPassword = username + ":" + password;
String userAndPasswordBase64 = Base64.encodeBase64String(userAndPassword.getBytes());
webClient.addRequestHeader("Proxy-Authorization", "Basic "+userAndPasswordBase64);

此Base64类来自Apache Commons Codec。

我使用以下方法传递端口和主机，但可能你的方式也很好。

webClient.getProxyConfig().setProxyHost(proxyHost);
webClient.getProxyConfig().setProxyPort(proxyPort);

Answer 2

虽然你没有把完整的堆栈跟踪放进去，但我猜这个错误正在被抛出：

final HtmlPage page = (HtmlPage)webClient.getPage(url);

这是因为getPage调用返回的是UnexpectedPage而不是HtmlPage。看一下documentation for UnexpectedPage，看来页面请求会返回一个不是text / html的Content-type，所以htmlunit不知道如何处理它。您应该打开调试并查看实际返回的内容以找出错误。

Answer 3

我无法使用HtmlUnit在代理服务器上执行NTLM身份验证。当我使用HttpClient（HtmlUnit构建在此之上）并使用NTLM身份验证设置代理设置。它工作。这是相同的代码。

HttpClient client = new HttpClient();
client.getHostConfiguration().setProxy("companyproxy.server", 8080);
List authPrefs = new ArrayList();
authPrefs.add(AuthPolicy.NTLM);

client.getState().setProxyCredentials(
    new AuthScope(null, 8080, null),
    new NTCredentials("username", "pwd", "", "DOMAIN"));

client.getParams().setParameter(AuthPolicy.AUTH_SCHEME_PRIORITY, authPrefs);

GetMethod method = new GetMethod(url);

method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, 
        new DefaultHttpMethodRetryHandler(3, false));

Answer 4

正如Rob所说，HtmlUnit无法检测到它是一个HTML页面。

请提供样本到用户列表，以便我们进一步调查

Answer 5

使用HTMLUnit 2.14，这对我有用：

    DefaultCredentialsProvider cp = (DefaultCredentialsProvider) client.getCredentialsProvider();
    cp.addNTLMCredentials(proxyUser, proxyPassword, proxyHost, proxyPort, null, domain);

使用HTMLUnit连接到URL时出现错误消息

5 个答案: