Question

我已经在Apple的应用程序商店中发布了一个移动应用程序。

此SPI客户端应用程序在服务器端使用Rest API来检索有关特定巴士站的巴士到达的实时信息。

该应用程序的工作时间长达6个月。

Rest API使用WebHarvest从网站中删除实际数据信息（例如：http://www.metlink.org.nz/stop/4912/departures）。

几天前，通过添加以下行改变了从我的服务器端代码中删除的HTML页面：

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">

从那以后，我的应用程序停止了工作。

我知道我可以使用regExp删除上面的行，但我想知道是否有办法通知WebHarvest禁用XML验证。禁用XML验证，我不需要进入我拥有的每个配置，并将我的xpath表达式更改为regExp以去除上面的行。

这是我的配置文件：

<config charset="UTF-8">

    <var-def name="pageContentStr">
        <html-to-xml>
            <http url="http://www.metlink.org.nz/stop/${stationID.toString()}/departures" />
        </html-to-xml>
    </var-def>

    <var-def name="serverTime">
        <xpath expression="/html/body/ul/li/span/text()">
            <var name="pageContentStr" />
        </xpath>
    </var-def>

    <var-def name="busRTI">
            <xpath expression="//tbody/tr[@data-code]/concat(td[1]/a[starts-with(@href,'timetables/')]/span/text(),'::',td[1]/a[starts-with(@href,'timetables/bus/')]/span/attribute::style,'::',td[2]/span/text(),'::',td[3]/span/text())">
                <var name="pageContentStr" />
            </xpath>
    </var-def>
</config>

如果我在WebHarvest GUI中运行它，那么上面插入的配置文件工作正常（很奇怪）。但是，在我的Rest API中运行它时收到错误。这是我收到的错误：

exception

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.webharvest.exception.ScraperXPathException: Error parsing XPath expression (XPath = [/html/body/ul/li/span/text()])!
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:948)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
root cause

org.webharvest.exception.ScraperXPathException: Error parsing XPath expression (XPath = [/html/body/ul/li/span/text()])!
    org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:70)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.processors.BodyProcessor.execute(BodyProcessor.java:25)
    org.webharvest.runtime.processors.VarDefProcessor.execute(VarDefProcessor.java:59)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.Scraper.execute(Scraper.java:166)
    org.webharvest.runtime.Scraper.execute(Scraper.java:179)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.scrapeBusesForStation(MetLinkAdapterImpl.java:147)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.getStationBuses(MetLinkAdapterImpl.java:118)
    com.didibaba.services.BusStationServiceImpl.getBusStationInfoByName(BusStationServiceImpl.java:80)
    com.didibaba.web.controllers.BusStationController.getBusStationInfo(BusStationController.java:36)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
root cause

net.sf.saxon.trans.XPathException: org.xml.sax.SAXParseExceptionpublicId: -//W3C//DTD HTML 4.0 Transitional//EN; systemId: `http://www.w3.org/TR/REC-html40/loose.dtd`; lineNumber: 31; columnNumber: 3; The declaration for the entity "HTML.Version" must end with '>'.
    net.sf.saxon.event.Sender.sendSAXSource(Sender.java:420)
    net.sf.saxon.event.Sender.send(Sender.java:169)
    net.sf.saxon.Configuration.buildDocument(Configuration.java:3346)
    net.sf.saxon.Configuration.buildDocument(Configuration.java:3288)
    net.sf.saxon.query.StaticQueryContext.buildDocument(StaticQueryContext.java:327)
    org.webharvest.utils.XmlUtil.evaluateXPath(XmlUtil.java:77)
    org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:68)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.processors.BodyProcessor.execute(BodyProcessor.java:25)
    org.webharvest.runtime.processors.VarDefProcessor.execute(VarDefProcessor.java:59)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.Scraper.execute(Scraper.java:166)
    org.webharvest.runtime.Scraper.execute(Scraper.java:179)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.scrapeBusesForStation(MetLinkAdapterImpl.java:147)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.getStationBuses(MetLinkAdapterImpl.java:118)
    com.didibaba.services.BusStationServiceImpl.getBusStationInfoByName(BusStationServiceImpl.java:80)
    com.didibaba.web.controllers.BusStationController.getBusStationInfo(BusStationController.java:36)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
root cause

org.xml.sax.SAXParseExceptionpublicId: -//W3C//DTD HTML 4.0 Transitional//EN; systemId: http://www.w3.org/TR/REC-html40/loose.dtd; lineNumber: 31; columnNumber: 3; The declaration for the entity "HTML.Version" must end with '>'.
    com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
    com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
    com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
    com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
    com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1388)
    com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.scanEntityDecl(XMLDTDScannerImpl.java:1562)
    com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.scanDecls(XMLDTDScannerImpl.java:1964)
    com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.scanDTDExternalSubset(XMLDTDScannerImpl.java:297)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1162)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1049)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:962)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
    com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:116)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:489)
    com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
    com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
    com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
    com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
    net.sf.saxon.event.Sender.sendSAXSource(Sender.java:396)
    net.sf.saxon.event.Sender.send(Sender.java:169)
    net.sf.saxon.Configuration.buildDocument(Configuration.java:3346)
    net.sf.saxon.Configuration.buildDocument(Configuration.java:3288)
    net.sf.saxon.query.StaticQueryContext.buildDocument(StaticQueryContext.java:327)
    org.webharvest.utils.XmlUtil.evaluateXPath(XmlUtil.java:77)
    org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:68)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.processors.BodyProcessor.execute(BodyProcessor.java:25)
    org.webharvest.runtime.processors.VarDefProcessor.execute(VarDefProcessor.java:59)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.Scraper.execute(Scraper.java:166)
    org.webharvest.runtime.Scraper.execute(Scraper.java:179)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.scrapeBusesForStation(MetLinkAdapterImpl.java:147)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.getStationBuses(MetLinkAdapterImpl.java:118)
    com.didibaba.services.BusStationServiceImpl.getBusStationInfoByName(BusStationServiceImpl.java:80)
    com.didibaba.web.controllers.BusStationController.getBusStationInfo(BusStationController.java:36)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

提前致谢。

Answer 1

您可以尝试在html-to-xml来电中使用omithtmlenvelope="true"。

<var-def name="pageContentStr">
    <html-to-xml omithtmlenvelope="true">
        <http url="http://www.metlink.org.nz/stop/${stationID.toString()}/departures" />
    </html-to-xml>
</var-def>

然而，不幸的是，正如你所说，我无法重现你所得到的错误，并且无法测试结果......

Answer 2

xpath评估程序抛出org.xml.sax.SAXParseException

时出现类似问题

White spaces are required between publicId and systemId.

如果您可以更改原始xml，则此问题的解决方案已经解决here。

Webharvest使用了html cleaner。我使用Complete Web-Harvest项目，因此可能会阻止在html-to-xml上添加doctype-tag。

我使用html清洁版2.6.1。并修改org.webharvest.runtime.processors.HtmlToXmlProcessor以支持这个更新的版本

HtmlCleaner cleaner = new HtmlCleaner( );
CleanerProperties cleanerProperties = cleaner.getProperties();

由于html清除支持omitDoctypeDeclaration - parameter完全省略了doctype，我补充说（将来可能通过使用scraper xml的额外属性来完成）。

cleanerProperties.setOmitDoctypeDeclaration(true);

希望它有所帮助，并且感谢webharvest的创建者，它是一个非常可靠的工具！

在WebHarvest中禁用XML验证

2 个答案: