我尝试使用HtmlUnit保存Google页面。但我无法获得适当的用户界面。当我检查保存的页面代码时,样式标签是空的。
我的代码在这里。
public static void main(String[] args) throws IOException {
FileUtils.cleanDirectory(new File("/home/user1/Documents/Aaa"));
WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setCssEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.waitForBackgroundJavaScriptStartingBefore(1000);
webClient.waitForBackgroundJavaScript(1000);
webClient.getOptions().setTimeout(5000);
System.out.println("******************loaded**********************************");
try {
HtmlPage page = webClient.getPage("https://www.google.com");
page.save(new File("/home/user1/Documents/Aaa/index.html"));
} catch (Exception e) {
System.out.println("******************catch***********************************");
e.printStackTrace();
}
webClient.close();
System.out.println("******************finished********************************");
}
我的页面看起来像
控制台日志
Dec 10, 2016 3:47:45 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'text/javascript'.
Dec 10, 2016 3:47:46 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError
SEVERE: runtimeError: message=[TypeError: object is not iterable] sourceName=[https://www.google.co.in/xjs/_/js/k=xjs.s.en.igGBAtxEWN0.O/m=sx,c,sb,cdos,cr,elog,hsm,jsa,r,qsm,j,p,d,csi/am=AAiUPF6wAOL_ISBuIRxBasDAoA/rt=j/d=1/t=zcms/rs=ACT90oGjQTdwqicso-l4vNE-7GeAqTtjtw] line=[10] lineSource=[null] lineOffset=[0]
Dec 10, 2016 3:47:46 PM com.gargoylesoftware.htmlunit.DefaultCssErrorHandler error
WARNING: CSS error: 'https://www.google.co.in/?gfe_rd=cr&ei=RtZLWL3cDsmL8QeW4Yb4AQ' [1:14018] Error in expression. (Invalid token " ". Was expecting one of: <NUMBER>, "inherit", <IDENT>, <STRING>, <HASH>, <EMS>, <EXS>, <LENGTH_PX>, <LENGTH_CM>, <LENGTH_MM>, <LENGTH_IN>, <LENGTH_PT>, <LENGTH_PC>, <ANGLE_DEG>, <ANGLE_RAD>, <ANGLE_GRAD>, <TIME_MS>, <TIME_S>, <FREQ_HZ>, <FREQ_KHZ>, <RESOLUTION_DPI>, <RESOLUTION_DPCM>, <PERCENTAGE>, <DIMENSION>, <URI>, <FUNCTION>, "progid:".)
Dec 10, 2016 3:47:46 PM com.gargoylesoftware.htmlunit.DefaultCssErrorHandler error
WARNING: CSS error: 'https://www.google.co.in/?gfe_rd=cr&ei=RtZLWL3cDsmL8QeW4Yb4AQ' [1:14042] Error in expression. (Invalid token " ". Was expecting one of: <NUMBER>, "inherit", <IDENT>, <STRING>, <HASH>, <EMS>, <EXS>, <LENGTH_PX>, <LENGTH_CM>, <LENGTH_MM>, <LENGTH_IN>, <LENGTH_PT>, <LENGTH_PC>, <ANGLE_DEG>, <ANGLE_RAD>, <ANGLE_GRAD>, <TIME_MS>, <TIME_S>, <FREQ_HZ>, <FREQ_KHZ>, <RESOLUTION_DPI>, <RESOLUTION_DPCM>, <PERCENTAGE>, <DIMENSION>, <URI>, <FUNCTION>, "progid:".)
Dec 10, 2016 3:47:46 PM com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine handleJavaScriptException
INFO: Caught script exception
======= EXCEPTION START ========
EcmaError: lineNumber=[551] column=[0] lineSource=[null] name=[TypeError] sourceName=[https://www.google.co.in/xjs/_/js/k=xjs.s.en.igGBAtxEWN0.O/m=sx,c,sb,cdos,cr,elog,hsm,jsa,r,qsm,j,p,d,csi/am=AAiUPF6wAOL_ISBuIRxBasDAoA/rt=j/d=1/t=zcms/rs=ACT90oGjQTdwqicso-l4vNE-7GeAqTtjtw] message=[TypeError: Cannot read property "Kf" from undefined (https://www.google.co.in/xjs/_/js/k=xjs.s.en.igGBAtxEWN0.O/m=sx,c,sb,cdos,cr,elog,hsm,jsa,r,qsm,j,p,d,csi/am=AAiUPF6wAOL_ISBuIRxBasDAoA/rt=j/d=1/t=zcms/rs=ACT90oGjQTdwqicso-l4vNE-7GeAqTtjtw#551)]
com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot read property "Kf" from undefined (https://www.google.co.in/xjs/_/js/k=xjs.s.en.igGBAtxEWN0.O/m=sx,c,sb,cdos,cr,elog,hsm,jsa,r,qsm,j,p,d,csi/am=AAiUPF6wAOL_ISBuIRxBasDAoA/rt=j/d=1/t=zcms/rs=ACT90oGjQTdwqicso-l4vNE-7GeAqTtjtw#551)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:921)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:852)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:824)
at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeEventListeners(EventListenersContainer.java:258)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeBubblingListeners(EventListenersContainer.java:322)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventTarget.fireEvent(EventTarget.java:206)
at com.gargoylesoftware.htmlunit.javascript.host.Window.dispatchEvent(Window.java:2033)
at com.gargoylesoftware.htmlunit.javascript.host.Window$2$1.run(Window.java:2119)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.host.Window$2.execute(Window.java:2124)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doProcessPostponedActions(JavaScriptEngine.java:966)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.access$500(JavaScriptEngine.java:101)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:916)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:852)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:824)
at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeEventListeners(EventListenersContainer.java:258)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeBubblingListeners(EventListenersContainer.java:322)
at com.gargoylesoftware.htmlunit.javascript.host.event.EventTarget.fireEvent(EventTarget.java:191)
at com.gargoylesoftware.htmlunit.html.DomElement$2.run(DomElement.java:1190)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.html.DomElement.fireEvent(DomElement.java:1195)
at com.gargoylesoftware.htmlunit.html.DomElement.fireEvent(DomElement.java:1163)
at com.gargoylesoftware.htmlunit.InteractivePage.setFocusedElement(InteractivePage.java:100)
at com.gargoylesoftware.htmlunit.InteractivePage.setFocusedElement(InteractivePage.java:66)
at com.gargoylesoftware.htmlunit.html.HtmlElement.detach(HtmlElement.java:1254)
at com.gargoylesoftware.htmlunit.html.DomNode.remove(DomNode.java:1132)
at com.gargoylesoftware.htmlunit.html.DomNode.replace(DomNode.java:1210)
at com.gargoylesoftware.htmlunit.javascript.host.dom.Node.replaceChild(Node.java:444)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:153)
at net.sourceforge.htmlunit.corejs.javascript.FunctionObject.call(FunctionObject.java:448)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1540)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.NativeArray.iterativeMethod(NativeArray.java:1694)
at net.sourceforge.htmlunit.corejs.javascript.NativeArray.execIdCall(NativeArray.java:405)
at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2575)
at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:321)
at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1540)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:794)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:906)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:803)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:779)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:975)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:352)
at com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:238)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doProcessPostponedActions(JavaScriptEngine.java:966)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.access$500(JavaScriptEngine.java:101)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:916)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:852)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:824)
at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptFunctionJob.runJavaScript(JavaScriptFunctionJob.java:52)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptExecutionJob.run(JavaScriptExecutionJob.java:102)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptJobManagerImpl.runSingleJob(JavaScriptJobManagerImpl.java:426)
at com.gargoylesoftware.htmlunit.javascript.background.DefaultJavaScriptExecutor.run(DefaultJavaScriptExecutor.java:157)
at java.lang.Thread.run(Thread.java:745)
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot read property "Kf" from undefined (https://www.google.co.in/xjs/_/js/k=xjs.s.en.igGBAtxEWN0.O/m=sx,c,sb,cdos,cr,elog,hsm,jsa,r,qsm,j,p,d,csi/am=AAiUPF6wAOL_ISBuIRxBasDAoA/rt=j/d=1/t=zcms/rs=ACT90oGjQTdwqicso-l4vNE-7GeAqTtjtw#551)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3915)
对于这种情况有什么想法吗?
被修改: 我认为错误是在CSS3过滤器属性时解析。在像这样的谷歌页面代码中。
filter:progid:DXImageTransform.Microsoft.gradient(startColorstr=#4387fd,endColorstr=#4683ea,GradientType=1)
在此过滤器中有一个冒号,而progrid有一个冒号。所以这会导致CSS Parsing错误。对此有何解决方案?
答案 0 :(得分:2)
起初:
WARNING: CSS error: 'https://www.google.co.in/?gfe_rd=cr&ei=hyxFWPuPJMyL8QfigajQCw'
这是一个警告,或多或少意味着css解析器不喜欢给定的css规则之一。这没有真正的影响,因为如果页面上的js代码询问样式的html元素,这可能会导致一些丢失的格式信息。
第二
webClient.waitForBackgroundJavaScriptStartingBefore(1000);
webClient.waitForBackgroundJavaScript(1000);
这两个调用不是配置选项 - 在给定位置执行这些调用是没有意义的。
但两者都只是小事,唯一真正的问题是
com.gargoylesoftware.htmlunit.ScriptException: Exception invoking constructor
异常。这通常会停止页面的js处理(但不是因为你将ThrowExceptionOnScriptError设置为false)。
要分析问题,我们至少需要构造函数抛出的异常的堆栈跟踪。通常,此堆栈跟踪也是日志的一部分。
但也许有一个更简单的解决方案。您尚未提供有关您正在使用的HtmlUnit版本的任何信息。我想建议使用最新的SNAPSHOT版本(你可以找到一些如何获得http://htmlunit.sourceforge.net/gettingLatestCode.html的信息)。我使用最新的htmlunit代码进行了快速测试,并且能够毫无例外地获取页面。
public static void main(String[] args) throws IOException {
FileUtils.cleanDirectory(new File("c://temp/htmlunit/aa"));
WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setCssEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setTimeout(5000);
System.out.println("******************loaded**********************************");
try {
HtmlPage page = webClient.getPage("https://www.google.com");
webClient.waitForBackgroundJavaScriptStartingBefore(1000);
webClient.waitForBackgroundJavaScript(1000);
page.save(new File("c://temp/htmlunit/aa/index.html"));
} catch (Exception e) {
System.out.println("******************catch***********************************");
e.printStackTrace();
}
webClient.close();
System.out.println("******************finished********************************");
}