如何编辑代码以遍历页面?

时间:2019-09-25 12:55:26

标签: python-3.x for-loop beautifulsoup python-requests

如何修复以下代码以遍历709页并返回所需的结果?

import pandas as pd
from bs4 import BeautifulSoup,Tag
import requests
import re

res=requests.get("https://cupcakemaps.com/cupcakes/cupcakes-near-me/p:2")
soup = BeautifulSoup(res.text,'lxml')
listings=soup.findAll(class_='media')
data = []
for listing in listings:
    listing_title=listing.find(True,{'title':True}).attrs['title']
    listing_Description=listing.find('p',{'class':'summary-desc'})

    if isinstance(listing_Description,Tag):
        listing_Description = listing_Description.text.strip()

    listing_address=listing.find('p',{'class':'contact-info'})

    if isinstance(listing_address,Tag):
        number_text = listing_address.text.strip()
        listing_address = ''.join(filter(str.isdigit,number_text))

    full_dict = {'Title': listing_title, 'Description': listing_Description, 'Address': listing_address}
    data.append(full_dict)

df = pd.DataFrame(data)
df.to_csv("contact.csv")
print(df)

我需要返回从页面1:https://cupcakemaps.com/cupcakes/cupcakes-near-me/到页面709:https://cupcakemaps.com/cupcakes/cupcakes-near-me/p:709

的所有709页面的结果。

2 个答案:

答案 0 :(得分:1)

使用format()函数。

for n in range(1,709):
    url="https://cupcakemaps.com/cupcakes/cupcakes-near-me/p:{}".format(n)
    print(url)

答案 1 :(得分:0)

import pandas as pd
from bs4 import BeautifulSoup,Tag
import requests
import re

data = []
s = "https://cupcakemaps.com/cupcakes/cupcakes-near-me/p:"
for x in range(1,709):
    res=requests.get(s+str(x))
    soup = BeautifulSoup(res.text,'lxml')
    listings=soup.findAll(class_='media')

    for listing in listings:
        listing_title=listing.find(True,{'title':True}).attrs['title']
        listing_Description=listing.find('p',{'class':'summary-desc'})

        if isinstance(listing_Description,Tag):
            listing_Description = listing_Description.text.strip()

        listing_address=listing.find('p',{'class':'contact-info'})

        if isinstance(listing_address,Tag):
            number_text = listing_address.text.strip()
            listing_address = ''.join(filter(str.isdigit,number_text))

        full_dict = {'Title': listing_title, 'Description': listing_Description, 'Address': listing_address}
        data.append(full_dict)

df = pd.DataFrame(data)
df.to_csv("contact.csv")
print(df)

未测试,因为我在代理后面,但是如果原始脚本有效,这应该可以工作...