Question

我想使用Regex从“确实”（仅第一页）中删除职位，职位和职位描述，并将结果存储到数据框中。这是链接：https://www.indeed.com/jobs?q=data+scientist&l=California

我已经使用BeautifulSoup完成了任务，他们工作得很好：

from urllib.request import urlopen
from bs4 import BeautifulSoup as BS
import pandas as pd

url = 'https://www.indeed.com/jobs?q=data+scientist&l=California'
htmlfile = urlopen(url)
soup = BS(htmlfile,'html.parser')

companies = []
locations = []
summaries = []

company = soup.findAll('span', attrs={'class':'company'})
for c in company:
    companies.append(c.text.replace("\n",""))

location = soup.findAll(class_ = 'location accessible-contrast-color-location')
for l in location:
    locations.append(l.text)

summary = soup.findAll('div', attrs={'class':'summary'})
for s in summary:
    summaries.append(s.text.replace("\n",""))

jobs_df = pd.DataFrame({'Company':companies, 'Location':locations, 'Summary':summaries})
jobs_df

BS的结果

    Company Location    Summary
0   Cisco Careers   San Jose, CA    Work on massive structured, unstru...
1   AllyO   Palo Alto, CA   Extensive knowledge of scientific ...
2   Driven Brands   Benicia, CA 94510   Develop scalable statistical, mach...
3   eBay Inc.   San Jose, CA    These problems require deep analys...
4   Disney Streaming Services   San Francisco, CA   Deep knowledge of machine learning...
5   Trimark Associates, Inc.    Sacramento, CA  The primary focus is in applying d...

但是当我尝试在正则表达式中使用相同的标签时，它失败了。

import urllib.request, urllib.parse, urllib.error
import re
import pandas as pd

url = 'https://www.indeed.com/jobs?q=data+scientist&l=California'
text = urllib.request.urlopen(url).read().decode()

companies = []
locations = []
summaries = []

company = re.findall('<span class="company">(.*?)</span>', text)
for c in company:
    companies.append(str(c))

location = re.findall('<div class="location accessible-contrast-color-location">(.*?)</div>', text)
for l in location:
    locations.append(str(l))

summary = re.findall('<div class="summary">(.*?)</div>', text)
for s in summary:
    summaries.append(str(s))

print(companies)
print(locations)
print(summaries)

有一个错误，说列表的长度不匹配，所以我检查了各个列表。事实证明无法提取内容。我从上面得到了什么：

[]
['Palo Alto, CA', 'Sunnyvale, CA', 'San Francisco, CA', 'South San Francisco, CA 94080', 'Pleasanton, CA 94566', 'Aliso Viejo, CA', 'Sacramento, CA', 'Benicia, CA 94510', 'San Bruno, CA']
[]

我做错了什么？

Answer 1

.匹配除换行符以外的任何字符。在HTML代码中，也有换行符。
因此，您需要在re.findall中使用 re.DOTALL 作为标志选项，如下所示：

company = re.findall('<span class="company">(.*?)</span>', text, flags=re.DOTALL)

从上面的代码中，您将不会仅获得名称。相反，您将获得所选择的span元素的所有后代。因此，您只需要选择所需的正则表达式部分即可。

for c in company:
  # selecting only the company name, discarding everything in the anchor tag.
  name = re.findall('<a.*>(.*)</a>', c, flags = re.DOTALL)
  for n in name:
    # doing a little cleanup by removing the newlines and spaces.
    companies.append(str(n.strip()))

print(companies)

输出：

['Driven Brands', 'Southern California Edison', 'Paypal', "Children's Hospital Los Angeles", 'Cisco Careers', 'University of California, Santa Cruz', 'Beyond Limits', 'Shutterfly', 'Walmart', 'Trimark Associates, Inc.']

关于位置和摘要，没有更多的HTML标记。
仅显示文字。

因此，只有re.DOTALL并剥离文本才能完成工作。
不需要第二个for循环和第二个findall。

Answer 2

for将匹配除行终止符之外的任何字符。您尝试获取的内容在新行.上。因此，您需要进行任何处理，包括行终止符。

您需要做的：\n

但是，这之后还需要进行一些清理。

如何使用Regex从HTML解析数据？

2 个答案: