Question

我正在浏览网页以获取表的行，然后将每一行追加到数据框。但是，我得到的列表不能被合并到一个数据帧中。如何转换此列表以允许pd.concat（）？

我尝试过pd.DataFrame(data)，但返回的是KeyError：0

这是来自print（data）https://imgur.com/a/t0v0QaU的结果：

[          Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price    $6,497    $8,311    $7,035,           Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price   $26,916   $27,175   $27,584,           Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price    $8,123    $8,022    $7,687,           Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price         —   $16,694   $21,842,           Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price   $13,888   $12,989   $13,314,           Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price   $28,095   $27,925   $28,406,           Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price    $7,242    $6,960    $8,436,           Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price   $25,839   $26,930   $26,710,           Unnamed: 0 2015-2016 2016-2017 2017-2018
0  Average net price   $18,603   $16,450   $17,145]

import pandas as pd
import requests
from bs4 import BeautifulSoup as bs

data = []
url = 'https://nces.ed.gov/collegenavigator/?id='
ids = pd.read_excel('ids.xlsx')
for index, row in ids.iterrows():
    try:
        r = requests.get(url+str(row[0]))
        soup = bs(r.content, 'lxml')
        table = pd.read_html(str(soup.select_one('table:has(td:contains("Average net price"))')))
        data.append(table[0])
    except:
        pass
print(data)

id是：

理想情况下，我希望输出具有一个id列以及每个年份范围（2015-2016、2016-2017等）的列，净价填充在矩阵中，如下所示：https://imgur.com/a/RC0hoGz < / p>

Answer 1

基本上，只需将ID保存在已解析数据帧的单独列中即可。现在它被忽略了

...
for index, row in ids.iterrows(): 
    try: 
        r = requests.get(url+str(row[0])) 
        soup = bs(r.content, 'lxml') 
        table = pd.read_html(str(soup.select_one('table:has(td:contains("Average net price"))')), index_col=0)[0] 
        table['id'] = row[0] # save the Id in a separate column
        data.append(table.set_index('id'))
    except: 
        pass

df = pd.concat(data)

结果：

       2015-2016 2016-2017 2017-2018
id                                  
180203    $6,497    $8,311    $7,035
222178   $26,916   $27,175   $27,584
138558    $8,123    $8,022    $7,687
412173         —   $16,694   $21,842
126182   $13,888   $12,989   $13,314
188429   $28,095   $27,925   $28,406
188438    $7,242    $6,960    $8,436
168528   $25,839   $26,930   $26,710
133872   $18,603   $16,450   $17,145

Answer 2

很酷的问题，

因此，当您将熊猫用于任何事物时，通常会为您提供系列或数据框作为输出。因此，当您创建一个名为data的列表，然后将table[0]附加到该列表时。您以为自己要附加一个列表（我认为）。但是pd.read_html给出了一个数据帧。因此，您只需要创建data作为数据框，然后将每个数据框附加到该数据框即可。

这是解决方法：

import pandas as pd
import requests
from bs4 import BeautifulSoup as bs

data = pd.DataFrame()
url = 'https://nces.ed.gov/collegenavigator/?id='
ids = pd.read_excel('ids.xlsx')
for index, row in ids.iterrows():
    try:
        r = requests.get(url+str(row[0]))
        soup = bs(r.content, 'lxml')
        table = pd.read_html(str(soup.select_one('table:has(td:contains("Average net price"))')))
        data = data.append(table[0], ignore_index=True)
    except:
        pass

希望有帮助。

Answer 3

使用：

sleep

输出：

import pandas as pd
import requests
from bs4 import BeautifulSoup as bs

data = []
url = 'https://nces.ed.gov/collegenavigator/?id='
ids = pd.read_excel('ids.xlsx')

for index, row in ids.iterrows():

    try:
        r = requests.get(url+str(row[0]))
        soup = bs(r.content, 'lxml')
        table = pd.read_html(str(soup.select_one('table:has(td:contains("Average net price"))')))
        dataframe=table[0]
        dataframe.index=row
        data.append(dataframe)
    except:
        pass


df_values= (pd.concat(data,sort=False)
              .drop('Unnamed: 0',axis=1)
              .rename_axis(index='UnitID') )
print(df_values)

如何将此列表转换为数据框？

3 个答案: