Question

我想知道如何使用漂亮的汤/请求从一个网站抓取多个不同的网页/城市，而不必一遍又一遍地重复我的代码。

这是我现在的代码：

Region = "Marrakech"
Spider = 20

def trade_spider(max_pages):
    page = -1

    partner_ID = 2
    location_ID = 25

    already_printed = set()

    while page <= max_pages:
        page += 1
        response = urllib.request.urlopen("http://www.jsox.com/s/search.json?q=" + str(Region) +"&page=" + str(page))
        jsondata = json.loads(response.read().decode("utf-8"))
        format = (jsondata['activities'])
        g_data = format.strip("'<>()[]\"` ").replace('\'', '\"')
        soup = BeautifulSoup(g_data)



        hallo = soup.find_all("article", {"class": "activity-card"})

        for item in hallo:
            headers = item.find_all("h3", {"class": "activity-card"})
            for header in headers:
                header_final = header.text.strip()
                if header_final not in already_printed:
                    already_printed.add(header_final)

            deeplinks = item.find_all("a", {"class": "activity"})
            for t in set(t.get("href") for t in deeplinks):
                deeplink_final = t
                if deeplink_final not in already_printed:
                    already_printed.add(deeplink_final)

            end_final = "Header: " + header_final + " | " + "Deeplink: " + deeplink_final
            print(end_final)

 trade_spider(int(Spider))

我的目标是从一个特定网站中理想地抓取多个城市/地区。

现在，我可以手动执行此操作，一遍又一遍地重复我的代码并抓取每个网站，然后将每个数据框的结果连接在一起，但这看起来非常单一。我想知道是否有人有更快的方式或任何建议？

我尝试在我的区域标记中添加第二个城市，但不起作用

Region = "Marrakech","London"

任何人都可以帮助我吗？任何反馈都表示赞赏。

Answer 1

Region = ["Marrakech","London"]

将while循环放在for循环中，然后将页面重置为-1。

for reg in Region:
   pages = -1

并在请求网址时用reg替换Region。

Region = ["Marrakech","London"]    
Spider = 20

def trade_spider(max_pages):

    partner_ID = 2
    location_ID = 25
    already_printed = set()
    for reg in Region:
        page = -1  
        while page <= max_pages:
            page += 1
            response = urllib.request.urlopen("http://www.jsox.com/s/search.json?q=" + str(reg) +"&page=" + str(page))
            jsondata = json.loads(response.read().decode("utf-8"))
            format = (jsondata['activities'])
            g_data = format.strip("'<>()[]\"` ").replace('\'', '\"')
            soup = BeautifulSoup(g_data)



            hallo = soup.find_all("article", {"class": "activity-card"})

            for item in hallo:
                headers = item.find_all("h3", {"class": "activity-card"})
                for header in headers:
                    header_final = header.text.strip()
                    if header_final not in already_printed:
                        already_printed.add(header_final)

                deeplinks = item.find_all("a", {"class": "activity"})
                for t in set(t.get("href") for t in deeplinks):
                    deeplink_final = t
                    if deeplink_final not in already_printed:
                        already_printed.add(deeplink_final)

                end_final = "Header: " + header_final + " | " + "Deeplink: " + deeplink_final
                print(end_final)
trade_spider(int(Spider))

如何从网站抓取多个页面/城市（BeautifulSoup，Requests，Python3）

1 个答案: