Question

是否可以使用get方法实现无限读取废料。例如，

http://www.justdial.com/Ahmedabad/Bearing-Dealers/ct-302676

当我们向下滚动时，

为每个页面提供以下链接

http://www.justdial.com/functions/ajxsearch.php?national_search=0&act=pagination&city=Ahmedabad&search=Bearing+Dealers&where=&catid=302676&psearch=&prid=&page=2&SID=&mntypgrp=0&toknbkt=&bookDate=&jdsrc=

http://www.justdial.com/functions/ajxsearch.php?national_search=0&act=pagination&city=Ahmedabad&search=Bearing+Dealers&where=&catid=302676&psearch=&prid=&page=3&SID=&mntypgrp=0&toknbkt=&bookDate=&jdsrc=

到目前为止，我的代码如下：

import requests

def readJustDial(c):
    hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
           'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
           'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
           'Accept-Encoding': 'none',
           'Accept-Language': 'en-US,en;q=0.8',
           'Connection': 'keep-alive'}    
    for i in range(1,10):
        url = 'http://www.justdial.com/functions/ajxsearch.php?national_search=0&act=pagination&city='+str(c)+'&search=Bearing+Dealers&where=&catid=302676&psearch=&prid=&page='+str(i)+'&SID=&mntypgrp=0&toknbkt=&bookDate=&jdsrc='
        page = requests.get(url,hdr)

def main(): #this is main function of this program
    allCities=["Ahmedabad","Hyderabad","Bangalore","Kolkata","Chennai","Mumbai","Delhi-NCR","Pune"]
    for city in allCities:
        readJustDial(city)
        #print(city)

if __name__ == "__main__":
    main()

还请建议我可以对现有代码进行任何更改。我只是在学习python，所以任何建议都会很好。

Answer 1

尝试模仿正常工作浏览器xhr请求附带的标头。您可以使用浏览器的开发人员工具查看这些标题（我使用chrome's）。当我查看请求时，我发现它与这些标头一起发送：

Accept:application/json, text/javascript, */*; q=0.01
Accept-Encoding:gzip, deflate, sdch
Accept-Language:he-IL,he;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Cookie:f5avrbbbbbbbbbbbbbbbb=BEKIPCFANCEHKADKNPJJJLHGCDKJOEEGKIIEPAAPHGEDJDNKFFBPCKEGMMIAECHOLECIMLJDAICKIFECEPMNKJNMKIDIMHPCOMHNNHMANENHHKEGMABPKFGKBAPGCHCJ; ppc=; PHPSESSID=bh34mlv2ba4gmgbntjtsjtt753; www=1712105664.20480.0000; _gat=1; scity=Ahmedabad; sarea=; dealBackCity=Ahmedabad; inweb_city=Ahmedabad; profbd=0; bdcheck=1; _ga=GA1.2.338746795.1480258713; tab=toprs; BDprofile=1; prevcatid=302676; view=lst_v; main_city=Ahmedabad
Host:www.justdial.com
Referer:http://www.justdial.com/Ahmedabad/Bearing-Dealers/ct-302676
User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36
X-Requested-With:XMLHttpRequest

尝试使用这些标头发送请求，但Cookie除外（通常它们只是临时工作）。

如果这也不起作用，您将需要cookie。您可以使用浏览器（例如使用selenium），也可以对网页或cookie进行一些逆向工程，并尝试编写一种获取工作cookie的方法。

使用requests.get方法python进行无限滚动

1 个答案: