Question

我正在使用PyQuery从Web处理大量文档。 PyQuery使用lxml来解析HTML文档。

事实上，很多文档都是无效的HTML。因此，lxml无法成功解析这些无效文档，这会阻止我进一步获取信息。并且经常引发以下异常：

    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
        self.runUntilCurrent()
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
        self._startRunCallbacks(result)
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/home/hxiao/hiit/crawl/crawl/spiders/basic.py", line 40, in parse
        doc = pq(response.body)
      File "/usr/local/lib/python2.7/dist-packages/pyquery/pyquery.py", line 226, in __init__
        elements = fromstring(context, self.parser)
      File "/usr/local/lib/python2.7/dist-packages/pyquery/pyquery.py", line 70, in fromstring
        result = getattr(lxml.html, meth)(context)
      File "/usr/local/lib/python2.7/dist-packages/lxml/html/__init__.py", line 706, in fromstring
        doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
      File "/usr/local/lib/python2.7/dist-packages/lxml/html/__init__.py", line 600, in document_fromstring
        value = etree.fromstring(html, parser, **kw)
      File "lxml.etree.pyx", line 3032, in lxml.etree.fromstring (src/lxml/lxml.etree.c:68121)

      File "parser.pxi", line 1786, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:102470)

      File "parser.pxi", line 1674, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:101299)

      File "parser.pxi", line 1074, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:96481)

      File "parser.pxi", line 582, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:91290)

      File "parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:92476)

      File "parser.pxi", line 631, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:91904)

    lxml.etree.XMLSyntaxError: line 649: htmlParseEntityRef: expecting ';'

我在问什么：

我想让lxml以不太严格的方式解析，以便可以忽略此无效性。

Answer 1

这个答案可能不是很有帮助，但我调查了类似的问题。

也许你可以看一下pyquery的这个提示？

http://pythonhosted.org/pyquery/tips.html

lxml / pyquery：以不太严格的方式解析

1 个答案: