Question

我正试图通过网络废除此页面https://steamcommunity.com/market/listings/730/Special%20Agent%20Ava%20%7C%20FBI/中列出的所有项目的价格我在请求模块中使用了python。我在TypeError: 'NoneType' object is not subscriptable行遇到results.update(jsonData2['listinginfo'])错误。仅当列出的项目数大于〜2000时，才会出现此错误。我想如果我尝试删除大量数据，该网站将阻止我。下面是我的代码

import requests
import math
import time

url = 'https://steamcommunity.com/market/listings/730/Special%20Agent%20Ava%20%7C%20FBI/render/'
results = {}
price_list = []
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'}
payload = {
    'query':'', 
    'start': '0',
    'count': '100',
    'country': 'US',
    'language': 'English',
    'currency': '1'}
r2 = requests.get(url='https://steamcommunity.com/', headers=headers)
cookiesDict = r2.cookies.get_dict()
headers.update({'Set-Cookie':'sessionid=' + cookiesDict['sessionid']})
#we get cookie for particular session id
r = requests.get(url=url, params=payload, headers=headers)
jsonData = r.json()
total_count = jsonData['total_count']
total_pages = math.ceil(total_count/100)
for page in range(0,total_pages):
    payload ={
        'query':'', 
        'start': '%s' %(page*100),
        'count': '100',
        'country': 'US',
        'language': 'English',
        'currency': '1'}
    jsonData2 = requests.get(url=url, params=payload, headers=headers).json()
    results.update(jsonData2['listinginfo'])
    print ('Aquired page %s of %s...' %(page, total_pages))
for i in results.items():
    price = (i['converted_price'] + i['converted_fee']) / 100
    price_list.append(price)
price_list.sort()
f = open('agentava.txt','a+')
for i in range(len(price_list)):
    f.write(str(price_list[i])+'\n')
f.close()

如果有一种方法可以获取所有数据，请共享。如果Steam网站阻止我获取数据的情况，那么建议我采取一种绕过该方法的方式（如果可能）。谢谢

TypeError：使用请求和Python进行网络抓取时，“ NoneType”对象无法下标

0 个答案: