Question

我正在试图抓住灰狗种族的信息。例如，我想要抓http://www.gbgb.org.uk/RaceCard.aspx?dogName=Hardwick%20Serena。这个页面显示了狗Hardwick Serena的所有结果，但它分成几页。

检查页面，它显示在“下一页”按钮下：

<input type="submit" name="ctl00$ctl00$mainContent$cmscontent$DogRaceCard$lvDogRaceCard$ctl00$ctl03$ctl01$ctl12" value=" " title="Next Page" class="rgPageNext">.

我希望有一个HTML链接，我可以用于下一次的刮擦，但没有运气。通过查看网络流量进一步检查，显示浏览器为__VIEWSTATE发送了一个可怕的长（哈希？）字符串。可能保护数据库？

我正在寻找一种方法来刮掉一只狗的所有页面，通过迭代所有页面，或者通过增加页面长度来显示第1页上的100多行。底层数据库是.aspx。

我正在使用Python 3.5和BeautifulSoup。

当前代码：

    import requests
    from   bs4 import BeautifulSoup

    url = 'http://www.gbgb.org.uk/RaceCard.aspx?dogName=Hardwick%20Serena'

    with requests.session() as s:
        s.headers['user-agent'] = 'Mozilla/5.0'

        r    = s.get(url)
        soup = BeautifulSoup(r.content, 'html5lib')

        target = 'ctl00$ctl00$mainContent$cmscontent$DogRaceCard$btnFilter_input'

        data = { tag['name']: tag['value'] 
            for tag in soup.select('input[name^=ctl00]') if tag.get('value')
        }
        state = { tag['name']: tag['value'] 
            for tag in soup.select('input[name^=__]')
        }

        data.update(state)

        numberpages = int(str(soup.find('div', 'rgWrap rgInfoPart')).split(' ')[-2].split('>')[1].split('<')[0])
        # for page in range(last_page + 1):

        for page in range(numberpages):
            data['__EVENTTARGET'] = target.format(page)
            #data['__VIEWSTATE'] = target.format(page)
            print(10)
            r    = s.post(url, data=data)
            soup = BeautifulSoup(r.content, 'html5lib')

            tables = soup.findChildren('table')
            my_table = tables[9]
            rows = my_table.findChildren(['th', 'tr'])

            tabel = [[]]
            for i in range(len(rows)):
                 cells = rows[i].findChildren('td')
                 tabel.append([])
                 for j in range(len(cells)):
                     value = cells[j].string
                     tabel[i].append(value)

            table = []
            for i in range(len(tabel)):
                if len(tabel[i]) == 16:
                    del tabel[i][-2:]
                    table.append(tabel[i])

Answer 1

在这种情况下，对于请求的每个页面，都会使用表单网址编码参数__EVENTTARGET＆amp;发出__VIEWSTATE个请求。 __VIEWSTATE：

input

__EVENTTARGET

<a href="javascript:__doPostBack('ctl00$ctl00$mainContent$cmscontent$DogRaceCard$lvDogRaceCard$ctl00$ctl03$ctl01$ctl07','')"> <span>2</span> </a>对于每个页面都不同，并且每个页面链接都会从javacript函数传递值，因此您可以使用正则表达式提取它：

from bs4 import BeautifulSoup
import requests
import re

# extract data from page
def extract_data(soup):
    tables = soup.find_all("div", {"class":"race-card"})[0].find_all("tbody")

    item_list = [
        (
            t[0].text.strip(), #date
            t[1].text.strip(), #dist
            t[2].text.strip(), #TP
            t[3].text.strip(), #StmHCP
            t[4].text.strip(), #Fin
            t[5].text.strip(), #By
            t[6].text.strip(), #WinnerOr2nd
            t[7].text.strip(), #Venue
            t[8].text.strip(), #Remarks
            t[9].text.strip(), #WinTime
            t[10].text.strip(), #Going
            t[11].text.strip(), #SP
            t[12].text.strip(), #Class
            t[13].text.strip()  #CalcTm
        )
        for t in (t.find_all('td') for t in tables[1].find_all('tr'))
        if t
    ]
    print(item_list)

session = requests.Session()

url = 'http://www.gbgb.org.uk/RaceCard.aspx?dogName=Hardwick%20Serena'

response = session.get(url)
soup = BeautifulSoup(response.content, "html.parser")

# get view state value
view_state = soup.find_all("input", {"id":"__VIEWSTATE"})[0]["value"]

# get all event target values
event_target = soup.find_all("div", {"class":"rgNumPart"})[0]
event_target_list = [
    re.search('__doPostBack\(\'(.*)\',', t["href"]).group(1)
    for t in event_target.find_all('a')
]

# extract data for the 1st page
extract_data(soup)

# extract data for each page except the first
for link in event_target_list[1:]:
    print("get page {0}".format(link))
    post_data = {
        '__EVENTTARGET': link,
        '__VIEWSTATE': view_state
    }
    response = session.post(url, data=post_data)
    soup = BeautifulSoup(response.content, "html.parser")
    extract_data(soup)

python脚本：

{{1}}

刮擦和解析多页（aspx）表

1 个答案: