我正在从网站上刮取一张表,并且在获取数据方面没有任何问题,但是在打印最终输出时遇到了问题。在我提供的代码中,它可以打印所有内容,如果在我的“ for”语句中打印,则不会出现任何问题(请参阅注释掉的打印命令)。但是,如果稍后在代码中的“ for”语句之外打印它,则只会得到第一行。我想将这段代码放在一个更大的项目中,该输出(以及其他输出)都在一封电子邮件中。如何显示整个输出?
我尝试将每个表格行追加到列表中,我认为我做错了,但是它只是一遍又一遍地打印同一行或打印第一行中的各个字母。
import requests
from bs4 import BeautifulSoup
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}
#print('Scraping Iowa Dept of Banking...')
url = 'https://www.idob.state.ia.us/bank/docs/applica/app_status.aspx'
r = requests.get(url, headers = headers)
soup = BeautifulSoup(r.text, 'html.parser')
mylist = []
for tr in soup.find_all('tr')[2:]:
tds = tr.find_all('td')
if len(tds[5].text) == 1:
edate = "NA"
else:
edate = ""
if len(tds[6].text) == 1:
loc = "NA"
else:
loc = ""
output5 = ("Bank: %s, City: %s, Type: %s, Effective Date: %s, Location: %s, Comment: %s \r\n" % (tds[0].text, tds[1].text, tds[2].text.replace(" ", ""), tds[5].text+edate, tds[6].text.replace(" ", "")+loc, tds[7].text))
global outputs5
outputs5 = output5
#print(outputs5) #The whole table prints if printed here
if outputs5 is None:
outputs5 = "No information available"
print(outputs5)
print(outputs5) #only prints the first line
答案 0 :(得分:2)
我想使用python库的pandas并将表提取出来并导入到csv中。
import pandas as pd
tables=pd.read_html("https://www.idob.state.ia.us/bank/docs/applica/app_status.aspx")
tables[1].to_csv('output.csv')
Csv看起来像这样。
安装熊猫非常容易。只需在命令提示符下键入。
pip安装熊猫
答案 1 :(得分:1)
尝试这样,您需要将输出附加到列表中,然后在打印之前将列表连接在一起。
循环中的打印有效的原因是因为它实际上打印了5次,而不仅仅是一次。
import requests
from bs4 import BeautifulSoup
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}
#print('Scraping Iowa Dept of Banking...')
url = 'https://www.idob.state.ia.us/bank/docs/applica/app_status.aspx'
r = requests.get(url, headers = headers)
soup = BeautifulSoup(r.text, 'html.parser')
mylist = []
for tr in soup.find_all('tr')[2:]:
tds = tr.find_all('td')
if len(tds[5].text) == 1:
edate = "NA"
else:
edate = ""
if len(tds[6].text) == 1:
loc = "NA"
else:
loc = ""
output5 = ("Bank: %s, City: %s, Type: %s, Effective Date: %s, Location: %s, Comment: %s \r\n" % (tds[0].text, tds[1].text, tds[2].text.replace(" ", ""), tds[5].text+edate, tds[6].text.replace(" ", "")+loc, tds[7].text))
mylist.append(outputs5)
if mylist == []:
print("No information available")
print('\n'.join(mylist))
答案 2 :(得分:1)
您是否尝试过使用熊猫的.read_html()
?
import pandas as pd
url ='https://www.idob.state.ia.us/bank/docs/applica/app_status.aspx'
tables = pd.read_html(url)
输出:
print (tables[-1].to_string())
Bank City Type Accepted Approved Effective # Location Comment
0 State Bank New Hampton Merge With and Into 05/21/2019 NaN NaN NaN Application to merge State Bank, New Hampton, ...
1 Farmers and Traders Savings Bank Douds Merge With and Into 05/20/2019 NaN NaN NaN Application to merge Farmers and Traders Savin...
2 City State Bank Norwalk Establish a Bank Office 05/15/2019 05/29/2019 NaN Mesa, AZ Application by City State Bank, Norwalk, to es...
3 Availa Bank Carroll Purchase and Assumption 04/16/2019 04/30/2019 NaN NaN Application by Availa Bank, Carroll, to purcha...
4 West Bank West Des Moines Establish a Bank Office 04/16/2019 05/02/2019 05/10/2019 Mankato, MN Application by West Bank, West Des Moines, to ...
5 West Bank West Des Moines Establish a Bank Office 04/10/2019 05/02/2019 05/07/2019 Owatonna, MN Application by West Bank, West Des Moines, to ...
6 West Bank West Des Moines Establish a Bank Office 04/09/2019 05/02/2019 05/07/2019 NaN Application by West Bank, West Des Moines, to ...
7 Iowa State Bank Algona Establish a Bank Office 03/15/2019 NaN NaN Phoenix, AZ Application by Iowa State Bank, Algona, to est...
8 Peoples Savings Bank Elma Merge With and Into 03/13/2019 04/24/2019 NaN NaN Application to merge Peoples State Bank, Elma,...
9 Two Rivers Bank & Trust Burlington Purchase and Assumption 01/25/2019 01/31/2019 05/31/2019 NaN Application by Two Rivers Bank & Trust, Burlin...
10 Westside State Bank Westside Establish a Bank Office 01/25/2019 02/06/2019 NaN Bellevue, NE Application by Westside State Bank, Westside, ...
11 Northwest Bank Spencer Relocate a Bank Office 11/29/2018 12/12/2018 NaN Ankeny Application by Northwest Bank, Spencer, to rel...
12 City State Bank Norwalk Establish a Bank Office 11/21/2018 12/12/2018 NaN Norwalk Application by City State Bank, Norwalk, to es...
13 First Security Bank and Trust Company Charles City Relocate a Bank Office 06/21/2018 06/29/2018 NaN Manly Application by First Security Bank and Trust C...
14 Lincoln Savings Bank Cedar Falls Establish a Bank Office 06/04/2018 06/25/2018 NaN Des Moines Application by Lincoln Savings Bank, Cedar Fal...
15 Raccoon Valley Bank Perry Establish a Bank Office 02/12/2018 03/02/2018 NaN Grimes Application by Raccoon Valley Bank, Perry, to ...
16 Community Savings Bank Edgewood Relocate a Bank Office 01/25/2018 01/25/2018 NaN Manchester Application by Community Savings Bank, Edgewoo...
17 Luana Savings Bank Luana Establish a Bank Office 06/05/2017 08/16/2017 NaN Norwalk Application by Luana Savings Bank, Luana, to e...
18 Fort Madison Financial Company Fort Madison Change of Ownership NaN 10/19/2017 NaN NaN Application for Linda Sue Baier, Fort Madison,...
19 Lincoln Bancorp Reinbeck Change of Ownership NaN 12/10/2018 NaN NaN Application for Lincoln Bancorp Employee Stock...
20 Emmetsburg Bank Shares, Inc. Emmetsburg Change of Ownership NaN 01/17/2019 NaN NaN Application for Charles and Maryanna Sarazine,...
21 Albrecht Financial Services, Inc. Norwalk Change of Control NaN 03/27/2019 05/10/2019 NaN Application for Dean L. Albrecht 2014 Family T...
22 Solon Financial, Inc. Solon Change of Ownership NaN 03/05/2019 NaN NaN Application for Cordelia A. Cosgrove, Bruce A....
23 How-Win Development Co. Cresco Change of Ownership NaN 03/28/2019 NaN NaN Application for John Scott Thomson, as trustee...
24 Lee Capital Corp. Fort Madison Change of Ownership NaN 04/15/2019 NaN NaN Application by Jean M. Humphrey, Kathleen A. M...
25 FNB BanShares, Inc. West Union Change of Ownership NaN 05/02/2019 NaN NaN Application for James L. Moss, individually an...
26 Old O'Brien Banc Shares, Inc. Sutherland Change of Ownership NaN 03/06/2019 NaN NaN Application for James J. Johnson and Colleen D...
27 Pella Financial Group, Inc. Pella Change of Control NaN 03/15/2019 NaN NaN Application for Pella Financial Group, Inc., P...
28 BANK Wapello Amend or Restate Articles of Incorporation NaN 05/07/2019 05/07/2019 NaN Restatement of Articles of Incorporation.
29 Security Agency, Inc. Decorah Change of Ownership NaN 11/28/2018 NaN NaN Application for the 2018 Grantor Trust FBO Rac...
30 Arendt's Inc. Montezuma Change of Ownership NaN 05/29/2019 NaN NaN Application for C. W. Bolen, Montezuma, indivi...