Question

因此，我打算在一个许可证代码列表上进行迭代后，从网站FSC Public Search中抓取2个表（不同格式）。我的问题是，因为我想要的两个表（产品数据和证书数据）采用2种不同的格式，所以我必须分别将它们抓取。例如，产品数据以网页上的常规"tr"格式，而证书数据则以"div"格式。

从我之前提出的问题中，我几乎解决了我的问题，并且可以在一系列许可代码上完全正常地检索证书数据（“ div”表格）。但是，我无法按需输出产品数据表。它没有显示5个许可证代码的产品数据，而是显示了5个第一个许可证代码的副本。我已经尝试过将这种抓取功能放入已定义的函数get_data_by_code中，但仍然无法获得我想要的格式，它只是CSV文件中的表格。

基本上，我不确定在函数/脚本中应包含此抓取内容，因此，感谢您提供任何输入。

df3 = pd.DataFrame()
df = pd.read_csv("MS_License_Codes.csv")
codes = df["License Code"]
data = [
        ('code', code),
        ('submit', 'Search'),
        ]
response = requests.post('https://info.fsc.org/certificate.php', data=data)
soup = BeautifulSoup(response.content, 'lxml')



def get_data_by_code(code):
    data = [
        ('code', code),
        ('submit', 'Search'),
    ]

    response = requests.post('https://info.fsc.org/certificate.php', data=data)
    soup = BeautifulSoup(response.content, 'lxml')


 #scraping the certificate data

    status = soup.find_all("label", string="Status")[0].find_next_sibling('div').text
    first_issue_date = soup.find_all("label", string="First Issue Date")[0].find_next_sibling('div').text
    last_issue_date = soup.find_all("label", string="Last Issue Date")[0].find_next_sibling('div').text
    expiry_date = soup.find_all("label", string="Expiry Date")[0].find_next_sibling('div').text
    standard = soup.find_all("label", string="Standard")[0].find_next_sibling('div').text


    return [code, status, first_issue_date, last_issue_date, expiry_date, standard]

# Just insert here output filename and codes to parse...
OUTPUT_FILE_NAME = 'Certificate_Data.csv'

df3 = pd.DataFrame()

with open(OUTPUT_FILE_NAME, 'w') as f:
    writer = csv.writer(f)
    for code in codes:
        print('Getting code# {}'.format(code))
        writer.writerow((get_data_by_code(code)))

##attempting to scrape the product data

        table = soup.find_all('table')[0] 
        df1, = pd.read_html(str(table))
        df3 = df3.append(df1) 

df3.to_csv('Product_Data.csv', index = False, encoding='utf-8')

编辑

因此，使用下面的代码，我可以获得最后许可证代码的产品数据的5份副本。.但是我仍然看不到为什么会出现这种情况

df3 = pd.DataFrame()
for code in codes:
    print('Getting code# {}'.format(code))
    response = requests.post('https://info.fsc.org/certificate.php', data=data)
    soup = BeautifulSoup(response.content, 'lxml')

    table = soup.find_all('table')[0] 
    df1, = pd.read_html(str(table))
    df3 = df3.append(df1)        

df3.to_csv('Product_Data.csv', index = False, encoding='utf-8')

编辑2

我一直在使用的示例代码：

codes = ['FSC-C001777', 'FSC-C124838' ,'FSC-C068163','FSC-C101537','FSC-C005776']

格式编辑

这是正确的表格格式，但是如您所见，它是第一个许可证代码中的信息重复5次，而不是唯一数据。

这是我想要的格式和信息，在这里一切正常：

Answer 1

对于您提供的codes，此简化方法就足够了。它只是直接从BeautifulSoup中提取必要的信息，而无需使用Pandas来尝试提取它：

from bs4 import BeautifulSoup
import requests
import csv

fieldnames_cert = ['Code', 'Status', 'First Issue Date', 'Last Issue Date', 'Expiry Date', 'Standard']
fieldnames_prod = ['Code', 'Product Type', 'Trade Name', 'Species', 'Primary Activity', 'Secondary Activity', 'Main Output Category']

codes = ['FSC-C001777', 'FSC-C124838', 'FSC-C068163', 'FSC-C101537', 'FSC-C005776']

with open('Certificate_Data.csv', 'wb') as f_output_cert, \
     open('Product_Data.csv', 'wb') as f_output_prod:

    csv_output_cert = csv.writer(f_output_cert)
    csv_output_cert.writerow(fieldnames_cert)

    csv_output_prod = csv.writer(f_output_prod)
    csv_output_prod.writerow(fieldnames_prod)

    for code in codes:
        print('Getting code# {}'.format(code))
        response = requests.post('https://info.fsc.org/certificate.php', data={'code' : code, 'submit' : 'Search'})
        soup = BeautifulSoup(response.content, 'lxml')

        # Extract the certificate data
        div_cert = soup.find('div', class_='certificatecl')
        csv_output_cert.writerow([code] + [div.text for div in div_cert.find_all('div')])

        # Extract the product data
        table = soup.find('h2', id='products').find_next_sibling('table')

        for tr in table.find_all('tr')[1:]:
            row = [td.get_text(strip=True).encode('utf-8') for td in tr.find_all('td')]
            csv_output_prod.writerow([code] + row)

这将产生Certificate_Data.csv，其中包含：

Code,Status,First Issue Date,Last Issue Date,Expiry Date,Standard
FSC-C001777,Valid,2009-04-01,2018-02-16,2019-04-01,FSC-STD-40-004 V3-0
FSC-C124838,Valid,2015-03-23,2015-03-23,2020-03-22,FSC-STD-40-004 V3-0
FSC-C068163,Valid,2010-03-01,2017-08-23,2022-08-22,FSC-STD-40-003 V2-1;FSC-STD-40-004 V3-0
FSC-C101537,Valid,2010-10-01,2013-11-28,2018-11-27,FSC-STD-40-003 V2-1;FSC-STD-40-004 V3-0
FSC-C005776,Valid,2007-07-17,2017-07-17,2022-07-16,FSC-STD-40-004 V3-0

并生成包含以下内容的Product_Data.csv

Code,Product Type,Trade Name,Species,Primary Activity,Secondary Activity,Main Output Category
FSC-C001777,W12 Indoor furnitureW12.4 Beds,,,Secondary Processor,Secondary Processor,FSC Mix
FSC-C124838,"W18 Other manufactured wood productsW18.4 Tools, tool bodies and handles",, Abies spp; Betula spp.; Fagus sylvatica L.; Hevea brasiliensis; Paulownia tomentosa (Thunb. ex Murr) Steud; Picea spp.; Populus spp.; Quercus spp; Schima wallichii (DC.) Korth.; Swietenia macrophylla; Tilia spp.; Ulmus spp.,brokers/traders with physical posession,,FSC Mix;FSC 100%;FSC Recycled
FSC-C068163,P2 Paper,,,brokers/traders with physical posession,Distributor/Wholesaler,FSC Mix;FSC 100%;FSC Recycled
FSC-C068163,P3 Paperboard,,,brokers/traders with physical posession,Distributor/Wholesaler,FSC Mix;FSC 100%;FSC Recycled
FSC-C101537,P8 Printed materials,,,Printing and related service,Secondary Processor,FSC Mix;FSC 100%;FSC Recycled
FSC-C101537,P7 Stationery of paper,,,Printing and related service,Secondary Processor,FSC Mix;FSC 100%;FSC Recycled
FSC-C005776,W12 Indoor furnitureW12.10 Cupboards and chests,"Outros produtos, (baú, quadro espelho, etc.)", Eucalyptus spp; Pinus spp.,Secondary Processor,,FSC Mix
FSC-C005776,W12 Indoor furnitureW12.7 Office furniture,"Produtos para escritório (escrivaninha, mesa, gaveteiros, etc.)", Eucalyptus spp; Pinus elliottii,Secondary Processor,,FSC Mix
FSC-C005776,W12 Indoor furnitureW12.12 Parts of furniture,"Partes de movéis, (peças de reposição)", Eucalyptus spp; Pinus taeda,Secondary Processor,,FSC Mix
FSC-C005776,W12 Indoor furnitureW12.4 Beds,Camas, Eucalyptus spp; Pinus taeda,Secondary Processor,,FSC Mix

从网页上抓取2种不同格式的表格-美丽汤

1 个答案: