Question

此脚本需要一直贯穿RI_page_urls.csv，然后贯穿RI_License_urls.csv的所有结果URL，并获取业务信息。

它将从RI_page_urls.csv中提取所有URL，但随后仅从RI_License_urls.csv中运行并打印100个URL中的第一个。在运行第二部分之前，需要帮助弄清楚如何使其等待第一部分完成。

我非常感谢您的帮助。

下面是RI_page_urls.csv的网址：

http://www.crb.state.ri.us/verify_CRB.php

和代码：

from bs4 import BeautifulSoup as soup
import requests as r
import pandas as pd
import re
import csv

#pulls lic# url
with open('RI_page_urls.csv') as f_input:
    csv_input = csv.reader(f_input)

    for url in csv_input:
        data = r.get(url[0])
        page_data = soup(data.text, 'html.parser')
        links = [r'www.crb.state.ri.us/' + link['href']
            for link in page_data.table.tr.find_all('a') if re.search('licensedetail.php', str(link))]

        df = pd.DataFrame(links)
        df.to_csv('RI_License_urls.csv', header=False, index=False, mode = 'a')
#Code Above works!

#need to pull table info from license url    
#this pulls the first record, but doesn't loop through the requests

with open('RI_License_urls.csv') as f_input_2:
    csv_input_2 = csv.reader(f_input_2)

    for url in csv_input_2:
        data = r.get(url[0])
        page_data = soup(data.text, 'html.parser')
        company_info = (' '.join(info.get_text(", ", strip=True).split()) for info in page_data.find_all('h9'))

        df = pd.DataFrame(info, columns=['company_info'])
        df.to_csv('RI_company_info.csv', index=False)

Answer 1

嗯，问题有点不清楚，代码也有几处错误

data = r.get(url[0])

应该是因为其网址以http或https而不是www开头

data = r.get("http://"+url[0])

在下面的代码中，

info的定义不是这样，我只是假设它应该是company_info

 company_info = (' '.join(info.get_text(", ", strip=True).split()) for info in page_data.find_all('h9'))

        df = pd.DataFrame(info, columns=['company_info'])

因此完整的代码是

from bs4 import BeautifulSoup as soup
import requests as r
import pandas as pd
import re
import csv

#pulls lic# url
with open('RI_page_urls.csv') as f_input:
    csv_input = csv.reader(f_input)

    for url in csv_input:
        data = r.get(url[0])
        page_data = soup(data.text, 'html.parser')
        links = [r'www.crb.state.ri.us/' + link['href']
            for link in page_data.table.tr.find_all('a') if re.search('licensedetail.php', str(link))]

        df = pd.DataFrame(links)
        df.to_csv('RI_License_urls.csv', header=False, index=False, mode = 'a')
#Code Above works!

#need to pull table info from license url    
#this pulls the first record, but doesn't loop through the requests

with open('RI_License_urls.csv') as f_input_2:
    csv_input_2 = csv.reader(f_input_2)
    with open('RI_company_info.csv','a',buffering=0) as companyinfofiledescriptor:
        for url in csv_input_2:
            data = r.get("http://"+url[0])
            page_data = soup(data.text, 'html.parser')
            company_info = (' '.join(info.get_text(", ", strip=True).split()) for info in page_data.find_all('h9'))

            df = pd.DataFrame(company_info, columns=['company_info'])
            df.to_csv(companyinfofiledescriptor, index=False)
            print(df)

For循环-运行1个循环直至完成，然后运行下一个循环python

1 个答案: