Question

我想使用this website从中提取数据，但是这个网站没有API来方便数据提取。因此，网站上有一个包含数据的“表格”，但该表分布在多个页面上，下载选项不支持“批量”下载。所以我想找到一种通过Python下载它的方法。
我发现我可以通过邮寄请求来做到这一点。我做的是我从第1页开始，然后单击“下一页”按钮并观看网络选项卡，这给了我：

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 200-399/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:36:40 GMT
ETag:W/"8ac0b-cFdEBUwfdiyTQm/gpJHYzQ"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311; _gat=1
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=200-399
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=200-399

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)
Name
genome/

要查看页面切换后的更改，我再次执行此操作（因此再次单击“下一页”按钮）。

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 400-599/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:44:24 GMT
ETag:W/"5dc73-ODopjbbyl5M2vUJvUGN0Gw"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=400-599
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=400-599

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)

因此，从第1页开始，在帖子请求期间发生了重大变化（在我看来） - ＆gt; 2和第2页 - ＆gt; 3是：

响应标题内容范围：项目200-399 / 45999 - ＆gt;内容范围：项目400-599 / 45999
请求标题范围： items = 200-399 - ＆gt;范围：项= 400-599
请求标题 X-Range： items = 200-399 - ＆gt; X-范围：项= 400-599

这就是网站所说的：

第1页：1 - 200 of 45999结果（但是0-199的帖子，未在上面显示）
第2页：201 - 400的45999结果（但是200-399的帖子，见上文）
第3页：401 - 600 of 45999结果（但帖子为400-599，见上文）

我有零关于阅读这些POST请求的知识。但我想应该有办法说：

requests.post(some url) and extract the total page numbers (or number of genomes and devide these by the max number per page, which is 200 see post requests).

然后像：

for page_numb in range(page_numbers):
    r = requests.post(some_url...... + page_numb)
    #extract the information from the table

最后问题:) 我无法弄清楚如何设置像

这样的POST请求

p1 = requests.post(https://www.patricbrc.org/api/genome/... 0-199)
p2 = requests.post(https://www.patricbrc.org/api/genome/... 200-399)
etc...

（我希望响应将包含由某些分隔符分隔的表格数据）

我试过的代码：

import requests
import json

url = 'https://www.patricbrc.org/api/genome/'
payload = {"eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)"}
headers = {
    "X-Range": "items=0-199",
    "Range": "items=0-199",
    "content-type": "application/json"}


r = requests.post(url, data=json.dumps(payload), headers=headers)
print(r)

此代码无效但我想它应该是这样的。
我知道我应该将有效负载发布为JSON，但我不知道如何将eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)转换为JSON，这是关键：值对。

Answer 1

这是一个源代码（它的丑陋代码，但它只是示例，我建议您使用request.Session执行此操作）：

import sys
import json
import requests

URL = 'https://www.patricbrc.org/view/Taxonomy/2'
API = 'https://www.patricbrc.org/api/genome/'

# we need to get a Cookie
response = requests.get(URL)
if response.status_code != 200:
    sys.exit('Cookie request failed')
cookie = response.headers['Set-Cookie'].split(';')[0]
etag   = response.headers['ETag']

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
    'Cookie'          : '{};'.format(cookie),
    'ETag'            : etag
}

data = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=data)
if response.status_code != 200:
    sys.exit('API request failed')
data = json.loads(response.text)

# test data
row200 = data[0]
print 'FAMILY     :', row200['family']
print 'GC_CONTENT :', row200['gc_content'] 
print 'GENUS      :', row200['genus']
print 'ORDER      :', row200['order']

test data块的输出：

FAMILY     : Eubacteriaceae
GC_CONTENT : 38.78
GENUS      : Eubacterium
ORDER      : Clostridiales

修改你不需要cookie，这也可以正常工作：

import json import requests API = 'https://www.patricbrc.org/api/genome/' # now with cookie request a data headers = { 'Host' : 'www.patricbrc.org', 'Accept' : 'application/javascript, application/json, application/json', 'Accept-Language' : 'en-US,en;q=0.5', 'Referer' : 'https://www.patricbrc.org/view/Taxonomy/2', 'X-Range' : 'items=200-399', # your pages 'Range' : 'items=200-399', # your pages 'Content-Type' : 'application/rqlquery+x-www-form-urlencoded', } query = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)' response = requests.post(API, headers=headers, data=query) data = json.loads(response.text)

用于访问网站上的表数据的POST查询（Python）

1 个答案: