使用python的隐藏部分的Web报废表

时间:2016-10-18 23:59:07

标签: python web-scraping beautifulsoup html-table

我正在尝试从此表中获取信息:

<table class="table4 table4-1 table4-1-1"><thead><tr><th class="estilo1">No</th><th class="estilo2">Si</th><!--                                                        <th><div class="contenedor-vinculos6"><a title="Ver más " class="vinculo-interrogacion" href="#">Más información</a></div></th>--></tr></thead><tbody><tr><td class="estilo1"><span class="estilo3">100%<span class="numero-voto">(15)</span></span><div class="grafica1 grafica1-desacuerdo"><div class="item-grafica" style="width: 100%;"/></div></div></td><td class="estilo2"><span class="estilo3">0%<span class="numero-voto">(0)</span></span><div class="grafica1 grafica1-deacuerdo"><div class="item-grafica" style="width: 0%;"/></div></div></td><td><span class="display-none">Más información</span></td></tr></tbody></table>

我在python3中执行以下操作:

req = Request('http://www.congresovisible.org/votaciones/10918/',headers=headers)
web_page = urlopen(req)
soup = BeautifulSoup(web_page.read(), 'html.parser')
table= soup.find_all('table', attrs={'class':'table4 table4-1 table4-1-1'})

这有效,但只显示表格的一部分,它排除了以后的所有内容:

<td class="estilo2"><span class="estilo3...)

这是输出

[<table class="table4 table4-1 table4-1-1"><thead><tr><th class="estilo1">No</th><th class="estilo2">Si</th><!--                                                        <th><div class="contenedor-vinculos6"><a title="Ver más " class="vinculo-interrogacion" href="#">Más información</a></div></th>--></tr></thead><tbody><tr><td class="estilo1"><span class="estilo3">100%<span class="numero-voto">(15)</span></span><div class="grafica1 grafica1-desacuerdo"><div class="item-grafica" style="width: 100%;"></div></div></td></tr></tbody></table>]

我怎样才能提取整张桌子?

1 个答案:

答案 0 :(得分:1)

实际上很容易解决。 html.parser没有很好地解析这种格式不正确的HTML。请改用更宽松的 html5lib。这对我有用:

import requests
from bs4 import BeautifulSoup

response = requests.get('http://www.congresovisible.org/votaciones/10918/')
soup = BeautifulSoup(response.content, 'html5lib')
table = soup.find_all('table', attrs={'class':'table4 table4-1 table4-1-1'})
print(table)

请注意,这需要安装html5lib个包:

pip install --upgrade html5lib

顺便说一下,lxml解析器也可以工作:

soup = BeautifulSoup(response.content, 'lxml')