Question

我想从this网站抓取数据。该表扩展到多个页面，但是url没有类似page=3的内容，所以我无法遍历各个页面来获取数据。

我是否有另一种方法可以在没有遍历各种页面的情况下抓取这样的数据？如何从python访问不同的页面以获取数据？

Answer 1

使用@SeanBreckenridge提供的直接JSON链接，您可以创建一个包含所有数据的CSV文件，如下所示：

import requests
import json
import csv

header = ['University', 'Department', 'Overall Pay', 'LW Ratio', 'Academic Year', 'Program Year', 'Comment', 'V1', 'V2', 'V3', 'V4']

with open('output.csv', 'w', newline='') as f_output:
    csv_output = csv.writer(f_output)
    csv_output.writerow(header)

    for index in range(30):
        url = 'http://www.phdstipends.com/data/{}'.format(index)
        print(url)
        r = requests.get(url)

        if len(r.text) < 20:
            break

        csv_output.writerows(json.loads(r.text)['data'])

哪个会开始：

University,Department,Overall Pay,LW Ratio,Academic Year,Program Year,Comment,V1,V2,V3,V4
University of California - Santa Barbara (UCSB),Chemical Engineering,"$31,125",1.27,2014-2015,1st,,"$31,250",,,$125
Pennsylvania State University (Penn State),Chemistry,"$20,500",1.11,2014-2015,2nd,,"$20,500",,,
University of California - San Diego (UCSD),Mathematics,"$20,500",0.87,2013-2014,4th,,,"$18,000","$2,500",

从多个页面刮取表格

1 个答案: