Question

我正在尝试从https://www.msamb.com/ApmcDetail/ArrivalPriceInfo网站抓取数据。

这是我要抓取的数据。因此，在突出显示的下拉选择框中，有148种商品。

到目前为止，我正在通过选择每种商品来手动复制数据。这需要大量的人工来提取数据。

因此，为了使其自动化，我开始使用Python。以下是我在Python（3.7.8）代码中使用的库。

硒
BeautifulSoup
熊猫

这是我的Python代码。

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

from selenium.webdriver.support.ui import Select
#from selenium.webdriver.common.by import By

driver = webdriver.Chrome(executable_path='G:/data/depend/chromedriver.exe')
driver.get('https://www.msamb.com/ApmcDetail/ArrivalPriceInfo/')

commodity = Select(driver.find_element_by_id("CommoditiesId"))

#able to select commodities by value
commodity.select_by_value('08005')

# Iterating over the all the commodity an fetching <td> element
for option in commodity.options:
    #print(option.text)
    soup = BeautifulSoup(option.text)
    print(soup)    
    rows = soup.select('tr')
    print(rows)
    for row in rows[1:]:
        td = row.find_all('td')
        print(td)
        APMC = td[0].text.strip()
        print(APMC)

在这里，我可以从下拉选择框中通过 id 等于 CommoditiesId 的商品来获取商品。

一旦获取了商品列表（148），我便试图解析为该特定商品获取的HTML表内容。在这里，我可以打印每次迭代的商品名称，但不能打印，但不能打印 APMC，品种，单位，数量，包装箱，包装箱，模态列数据。 / p>

如果上述解决方案成立，我希望以~|~分隔格式输出，并希望添加两列，即日期，商品。因此，示例输出将如下所示（到目前为止，手动在数据文件下方进行准备）。

Date~|~Commodity~|~APMC~|~Variety~|~Unit~|~Quantity~|~Lrate~|~Hrate~|~Modal
    2020-07-11~|~APPLE~|~KOLHAPUR~|~QUINTAL~|~17~|~8500~|~14500~|~11500
    2020-07-11~|~APPLE~|~CHANDRAPUR-GANJWAD~|~QUINTAL~|~9~|~15000~|~17000~|~16000
    2020-07-11~|~APPLE~|~NASHIK~|~DILICIOUS- No.1~|~QUINTAL~|~60~|~9500~|~16000~|~13000
    2020-07-11~|~AMBAT CHUKA~|~PANDHARPUR~|~~|~NAG~|~7~|~10~|~10~|~10
    2020-07-10~|~AMBAT CHUKA~|~PUNE-MANJRI~|~~|~NAG~|~400~|~3~|~6~|~4
    2020-07-10~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~1300~|~4~|~5~|~4

Answer 1

您可以将它们保存到txt文件中，并且可以执行类似df = pd.read_csv("out.txt",delimiter='~|~')或

date = df['Date'] commodity = df['Commodity']

您可以将apmc附加到列表中，并在末尾读取数据框。

Answer 2

此脚本将遍历所有页面并将其保存到标准csv和~|~分隔的文本文件中：

import requests
import numpy as np
import pandas as pd
from bs4 import BeautifulSoup


url = 'https://www.msamb.com/ApmcDetail/ArrivalPriceInfo'
detail_url = 'https://www.msamb.com/ApmcDetail/DataGridBind?commodityCode={code}&apmcCode=null'
headers = {'Referer': 'https://www.msamb.com/ApmcDetail/ArrivalPriceInfo'}

soup = BeautifulSoup(requests.get(url).content, 'html.parser')
values = [(o['value'], o.text) for o in soup.select('#CommoditiesId option') if o['value']]

all_data = []
for code, code_name in values:
    print('Getting info for code {} {}'.format(code, code_name))
    soup = BeautifulSoup(requests.get(detail_url.format(code=code), headers=headers).content, 'html.parser')

    current_date = ''
    for row in soup.select('tr'):
        if row.select_one('td[colspan]'):
            current_date = row.get_text(strip=True)
        else:
            row = [td.get_text(strip=True) for td in row.select('td')]
            all_data.append({
                'Date': current_date,
                'Commodity': code_name,
                'APMC': row[0],
                'Variety': row[1],
                'Unit': row[2],
                'Quantity': row[3],
                'Lrate': row[4],
                'Hrate': row[5],
                'Modal': row[6],
            })

df = pd.DataFrame(all_data)
print(df)
df.to_csv('data.csv')                                       # <-- saves standard csv
np.savetxt('data.txt', df, delimiter='~|~', fmt='%s')       # <-- saves .txt file with '~|~' delimiter

打印：

...

Getting info for code 08071 TOMATO
Getting info for code 10006 TURMERIC
Getting info for code 08075 WAL BHAJI
Getting info for code 08076 WAL PAPDI
Getting info for code 08077 WALVAD
Getting info for code 07011 WATER MELON
Getting info for code 02009 WHEAT(HUSKED)
Getting info for code 02012 WHEAT(UNHUSKED)
            Date        Commodity          APMC Variety     Unit Quantity Lrate Hrate Modal
0     18/07/2020      AMBAT CHUKA    PANDHARPUR    ----      NAG       50     5     5     5
1     16/07/2020      AMBAT CHUKA    PANDHARPUR    ----      NAG       50     5     5     5
2     15/07/2020      AMBAT CHUKA    PANDHARPUR    ----      NAG      100     9     9     9
3     13/07/2020      AMBAT CHUKA    PANDHARPUR    ----      NAG       16     7     7     7
4     13/07/2020      AMBAT CHUKA          PUNE   LOCAL      NAG     2400     4     7     5
...          ...              ...           ...     ...      ...      ...   ...   ...   ...
4893  12/07/2020    WHEAT(HUSKED)        SHIRUR   No. 2  QUINTAL        2  1400  1400  1400
4894  17/07/2020  WHEAT(UNHUSKED)  SANGLI-MIRAJ    ----  QUINTAL      863  4000  4600  4300
4895  16/07/2020  WHEAT(UNHUSKED)  SANGLI-MIRAJ    ----  QUINTAL      475  4000  4500  4250
4896  15/07/2020  WHEAT(UNHUSKED)  SANGLI-MIRAJ    ----  QUINTAL      680  3900  4400  4150
4897  13/07/2020  WHEAT(UNHUSKED)  SANGLI-MIRAJ    ----  QUINTAL     1589  3900  4450  4175

[4898 rows x 9 columns]

保存data.txt：

0~|~18/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~50~|~5~|~5~|~5
1~|~16/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~50~|~5~|~5~|~5
2~|~15/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~100~|~9~|~9~|~9
3~|~13/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~16~|~7~|~7~|~7
4~|~13/07/2020~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~2400~|~4~|~7~|~5
5~|~12/07/2020~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~1700~|~3~|~8~|~5
6~|~19/07/2020~|~APPLE~|~KOLHAPUR~|~----~|~QUINTAL~|~3~|~9000~|~14000~|~11500
7~|~18/07/2020~|~APPLE~|~KOLHAPUR~|~----~|~QUINTAL~|~12~|~8500~|~15000~|~11750
8~|~18/07/2020~|~APPLE~|~NASHIK~|~DILICIOUS- No.1~|~QUINTAL~|~110~|~9000~|~16000~|~13000
9~|~18/07/2020~|~APPLE~|~SANGLI-PHALE BHAJIPALAM~|~LOCAL~|~QUINTAL~|~8~|~12000~|~16000~|~14000
10~|~17/07/2020~|~APPLE~|~MUMBAI-FRUIT MARKET~|~----~|~QUINTAL~|~264~|~9000~|~12000~|~10500
...

来自LibreOffice的csv文件的屏幕截图：

如何抓取HTML表格格式的数据？

2 个答案: