Question

我正在尝试抓取多个网址页面。

但是我只能抓第一页是否有办法获取所有页面。

这是我的代码。

from bs4 import BeautifulSoup as Soup 
import urllib, requests, re, pandas as pd

pd.set_option('max_colwidth',500)    # to remove column limit (Otherwise, we'll lose some info) 
df = pd.DataFrame()

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:
        comp_name = elem.find('span', attrs={'class':'company'}).getText().strip()
        job_title = elem.find('a', attrs={'class':'turnstileLink'}).attrs['title']
        home_url = "http://www.indeed.com"
        job_link = "%s%s" % (home_url,elem.find('a').get('href'))
        job_addr = elem.find('span', attrs={'class':'location'}).getText()
        date_posted = elem.find('span', attrs={'class': 'date'}).getText()
        description = elem.find('span', attrs={'class': 'summary'}).getText().strip()


        comp_link_overall = elem.find('span', attrs={'class':'company'}).find('a')
        if comp_link_overall != None:
        comp_link_overall = "%s%s" % (home_url, comp_link_overall.attrs['href'])
        else: comp_link_overall = None

        df = df.append({'comp_name': comp_name, 'job_title': job_title,
                'job_link': job_link, 'date_posted': date_posted,
                'overall_link': comp_link_overall, 'job_location': job_addr, 'description': description
                }, ignore_index=True)


df

df.to_csv('path\\web_scrape_Indeed.csv', sep=',', encoding='utf-8')

无论如何，请建议。

Answer 1

案例1：此处提供的代码正是您所拥有的

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

for elem in targetElements:

这里的问题是targetElements在第一个for循环中的每次迭代都会发生变化。

为避免这种情况，请将第二个for循环缩进第一个内容，如下所示：

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:

案例2：您的错误不是由于不正确的缩进造成的 （即不像原始帖子中的内容） 如果您的代码是正确的，那么targetElements可能就是一个空列表。这意味着target.findAll('div', class_ =' row result')不会返回任何内容。在这种情况下，访问网站，检查dom，然后修改你的抓取程序。

用美丽的汤刮多页

1 个答案: