Question

我正尝试使用以下脚本从webpage获取一些json响应。以下是在该网站中填充结果的步骤。点击位于此webpage底部的同意按钮，然后点击 EDIT SEARCH 按钮，最后在 SHOW RESULTS 按钮上保持不变。

我曾经这样尝试过：

import requests
from bs4 import BeautifulSoup

url = 'http://finra-markets.morningstar.com/BondCenter/Results.jsp'
post_url = 'http://finra-markets.morningstar.com/bondSearch.jsp'

payload = {
    'postData': {'Keywords':[]},
    'ticker': '',
    'startDate': '',
    'endDate': '',
    'showResultsAs': 'B',
    'debtOrAssetClass': '1,2',
    'spdsType': ''
}

payload_second = {
    'count': '20',
    'searchtype': 'B',
    'query': {"Keywords":[{"Name":"debtOrAssetClass","Value":"3,6"},{"Name":"showResultsAs","Value":"B"}]},
    'sortfield': 'issuerName',
    'sorttype': '1',
    'start': '0',
    'curPage': '1'
}

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36'
    s.headers['Referer'] = 'http://finra-markets.morningstar.com/BondCenter/UserAgreement.jsp'
    r = s.post(url,json=payload)
    s.headers['Access-Control-Allow-Headers'] = r.headers['Access-Control-Allow-Headers']
    s.headers['cf-request-id'] = r.headers['cf-request-id']
    s.headers['CF-RAY'] = r.headers['CF-RAY']
    s.headers['X-Requested-With'] = 'XMLHttpRequest'
    s.headers['Origin'] = 'http://finra-markets.morningstar.com'
    s.headers['Referer'] = 'http://finra-markets.morningstar.com/BondCenter/Results.jsp'
    r = s.post(post_url,json=payload_second)
    print(r.content)

这是我运行上面的脚本时得到的结果：

b'\n\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\n\n\n{}'

如何使脚本填充该站点的预期结果？

P.S。我不希望硒来完成这项工作。

Answer 1

http://finra-markets.morningstar.com/BondCenter/Results.jsp的响应不包含搜索结果。它必须异步获取数据。

找出哪些网络请求返回了搜索结果的一种简单方法是使用Firefox的Dev Tools在请求中搜索其中一个搜索结果：

要将HTTP请求转换为Python请求，我将请求从Firefox复制为CURL代码方法，将其导入Postman，然后将其导出为Python代码（我知道有点long（而且很懒））！）：

所有这些都导致以下代码：

import requests

url = "http://finra-markets.morningstar.com/bondSearch.jsp"

payload = "count=20&searchtype=B&query=%7B%22Keywords%22%3A%5B%7B%22Name%22%3A%22debtOrAssetClass%22%2C%22Value%22%3A%223%2C6%22%7D%2C%7B%22Name%22%3A%22showResultsAs%22%2C%22Value%22%3A%22B%22%7D%5D%7D&sortfield=issuerName&sorttype=1&start=0&curPage=1"
headers = {
    'User-Agent': "...",
    'Accept': "text/plain, */*; q=0.01",
    'Accept-Language': "en-US,en;q=0.5",
    'Content-Type': "application/x-www-form-urlencoded; charset=UTF-8",
    'X-Requested-With': "XMLHttpRequest",
    'Origin': "http://finra-markets.morningstar.com",
    'DNT': "1",
    'Connection': "keep-alive",
    'Referer': "http://finra-markets.morningstar.com/BondCenter/Results.jsp",
    'Cookie': "...",
    'cache-control': "no-cache"
    }

response = requests.request("POST", url, data=payload, headers=headers)

print(response.text)

响应不是100％JSON。因此，我只去除了外部空白和{B:..}部分：

>>> text = response.text.strip()[3:-1]
>>> import json
>>> data = json.loads(text)
>>> data['Columns'][0]                                                                                                             
{'moodyRating': {'ratingText': '', 'ratingNumber': 0},
 'fitchRating': {'ratingText': None, 'ratingNumber': None},
 'standardAndPoorRating': {'ratingText': '', 'ratingNumber': 0},

无法从发布发布请求的站点获取预期结果

1 个答案: