Question

我是网络爬虫的新手，可以使用一些帮助。我想使用Xpath抓取NBA的首发阵容，球队和球员的位置。我只是从名字开始，因为我遇到了问题。

到目前为止，这是我的代码：

from urllib.request import urlopen
from lxml.html import fromstring 


url = "https://www.lineups.com/nba/lineups"

content = str(urlopen(url).read())
comment = content.replace("-->","").replace("<!--","")
tree = fromstring(comment)


for nba, bball_row in enumerate(tree.xpath('//tr[contains(@class,"t-content")]')):
    names = bball_row.xpath('.//span[@_ngcontent-c5="long-player-name"]/text()')[0]
    print(names)

看起来程序运行没有错误，但名称未打印。任何有关如何更有效地使用Xpath进行解析的技巧将不胜感激。我尝试将Xpath帮助程序和Xpath Finder弄混。也许有一些技巧可以使过程更容易。预先感谢您的时间和精力！

Answer 1

位于script节点内的必需内容，看起来像

<script nonce="STATE_TRANSFER_TOKEN">window['TRANSFER_STATE'] = {...}</script>

您可以尝试执行以下操作以将数据提取为简单的Python字典：

import re
import json
import requests

source = requests.get("https://www.lineups.com/nba/lineups").text
dictionary = json.loads(re.search(r"window\['TRANSFER_STATE'\]\s=\s(\{.*\})<\/script>", source).group(1))

（可选）：粘贴dictionary here的输出，然后单击“美化”以将数据视为可读的JSON

然后您可以通过键访问所需的值，例如

for player in dictionary['https://api.lineups.com/nba/fetch/lineups/gateway']['data'][0]['home_players']:
    print(player['name'])

Kyrie Irving
Jaylen Brown
Jayson Tatum
Gordon Hayward
Al Horford

for player in dictionary['https://api.lineups.com/nba/fetch/lineups/gateway']['data'][0]['away_players']:
    print(player['name'])

D.J. Augustin
Evan Fournier
Jonathan Isaac
Aaron Gordon
Nikola Vucevic

更新

我想我只是把它弄得太复杂了：）

它应该像下面这样简单

import requests

source = requests.get("https://api.lineups.com/nba/fetch/lineups/gateway").json()
for player in source['data'][0]['away_players']:
        print(player['name'])

更新2

要获取所有球队的阵容，请使用以下

：

import requests

source = requests.get("https://api.lineups.com/nba/fetch/lineups/gateway").json()

for team in source['data']:
    print("\n%s players\n" % team['home_route'].capitalize())
    for player in team['home_players']:
        print(player['name'])
    print("\n%s players\n" % team['away_route'].capitalize())
    for player in team['away_players']:
        print(player['name'])

如何通过网络抓取NBA的首发阵容？

1 个答案: