我正在尝试将this table引入熊猫DataFrame中。我尝试使用熊猫read_html，尝试使用请求和bs4。我想抓住整个表格，但是在html代码中，表格分为3个块。不过，我还没有弄清楚如何确定它们中的每一个。
这是一个起始代码：
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = 'http://www2.bmf.com.br/pages/portal/bmfbovespa/lumis/lum-sistema-pregao-enUS.asp'
response = requests.get(url, params={'Data': '08/01/2018', 'Mercadoria': 'DI1'})
soup = BeautifulSoup(response.text, 'html.parser')
答案 0 :(得分：1)
可以帮助您解决的一个重要问题是，您在requests处获得的响应实际上并不包含呈现的table元素，但肯定包含所需的数据。
问题是该页面需要javascript才能呈现表。您可能会注意到您的数据在script元素内：
<script type="text/javascript">
var MercFut0 = "";
var MercFut1 = "";
var MercFut2 = "";
var MercFut3 = "";
MercFut0 = MercFut0 + '<table class="secondary">';
MercFut0 = MercFut0 + '<tr><td></td></tr>';
MercFut1 = MercFut1 + '<table class="secondary" id="teste">';
MercFut1 = MercFut1 + '<tr style="height: 120px;">';
MercFut2 = MercFut2 + '<table class="secondary" id="teste">';
MercFut2 = MercFut2 + '<tr style="height: 120px;">';
MercFut2 = MercFut2 + '<th class="text-center">Open Interest opening*</th>';
MercFut2 = MercFut2 + '<th class="text-center">Open Interest closing**</th>';
...
MercFut1 = MercFut1 + '</tr>';
MercFut0 = MercFut0 + '</table>';
MercFut1 = MercFut1 + '</table>';
MercFut2 = MercFut2 + '</table>';
MercFut3 = MercFut3 + '</table>';
MercadoFut0.innerHTML = MercFut3;
MercadoFut1.innerHTML = MercFut0;
MercadoFut2.innerHTML = MercFut1;
tableShow(2,false);
tableShow(9,false);
</script>
这时，最简单的方法可能是使用selenium之类的smth来简单地呈现此页面。
或者，您可以尝试获取此脚本并使用例如pyexecjs来执行它。
遵循以下原则：
import execjs
import requests
from bs4 import BeautifulSoup
import pandas as pd
url = 'http://www2.bmf.com.br/pages/portal/bmfbovespa/lumis/lum-sistema-pregao-enUS.asp'
response = requests.get(url, params={'Data': '08/01/2018', 'Mercadoria': 'DI1'})
soup = BeautifulSoup(response.text, 'html.parser')
# compile the desired table html from the data
script = soup.find("script", text=lambda text: text and 'tableShow' in text and "<table" in text).get_text()
script = """
var MercadoFut0 = {},
MercadoFut1 = {},
MercadoFut2 = {};
var tableShow = function () {};
function getTables() {
%s
return [MercFut1, MercFut2, MercFut3];
}
""" % script
ctx = execjs.compile(script)
table1, table2, table3 = ctx.call("getTables")
# parse tables into dataframes
df1 = pd.read_html(table1)[0]
df2 = pd.read_html(table2)[0]
df3 = pd.read_html(table3)[0]
print(df1)
print(df2)
print(df3)
然后，您可以“ concat” df1和df2，这将为您提供所需的表：
df = pd.concat([df2, df1], axis=1)
pd.set_option('display.expand_frame_repr', False)
print(df)
打印：
0 1 2 3 4 0 1 2 3 4 5 6 7 8 9 10
0 Open Interest opening* Open Interest closing** Number of Trades Trading Volume Financial Volume (R$) Previous Settlement*** Indexed Settlement**** Opening Price Minimum Price Maximum Price Average Price Last Price Settlement Price Change Last Bid Last Offer
1 442761 442761 0 0 0 99999.99 - 0.000 0.000 0.000 0.000 0.000 100000.00 0.01+ 0.000 0.000
2 760332 792464 147 114160 11351487370 99434.83 99434.83 6.404 6.403 6.410 6.406 6.407 99434.72 0.11- 6.404 6.408
3 2343218 2377609 183 99890 9885888562 98967.40 98967.40 6.429 6.421 6.435 6.423 6.425 98967.53 0.13+ 0.000 6.425
...
38 72923 74133 280 3920 126375887 32296.91 32296.91 11.510 11.450 11.600 11.521 11.580 32063.31 233.60- 11.590 11.610
39 5325 5325 0 0 0 28878.71 28878.71 0.000 0.000 0.000 0.000 0.000 28649.55 229.16- 0.000 11.680