Question

我的目标是在可汗学院的多个个人资料页面上抓取一些特定数据。并将数据放在csv文件中。

以下是刮取一个特定配置文件页面并将其放置在csv上的代码：

from bs4 import BeautifulSoup
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.khanacademy.org/profile/DFletcher1990/')
r.html.render(sleep=5)

soup=BeautifulSoup(r.html.html,'html.parser')

user_info_table=soup.find('table', class_='user-statistics-table')

dates,points,videos=[tr.find_all('td')[1].text for tr in user_info_table.find_all('tr')]

user_socio_table=soup.find_all('div', class_='discussion-stat')

data = {}
for gettext in user_socio_table:
   category = gettext.find('span')
   category_text = category.text.strip()
   number = category.previousSibling.strip()
   data[category_text] = number

filename = "khanscraptry1.csv"
f = open(filename, "w")
headers = "date, points, videos, questions, votes, answers, flags, project_request, project_replies, comments, tips_thx\n"
f.write(headers)
f.write(dates + "," + points.replace("," , "") + "," + videos + "," + data['questions'] + "," + data['votes'] + "," + data['answers'] + "," + data['flags raised'] + "," + data['project help requests'] + "," + data['project help replies'] + "," + data['comments'] + "," + data['tips and thanks'] + "\n")
f.close()

此代码与此特定链接（'https://www.khanacademy.org/profile/DFletcher1990/'）配合良好。

现在，当我将链接更改为可汗学院的其他个人资料时，例如：'https://www.khanacademy.org/profile/Kkasparas/'

我收到此错误：

KeyError: 'project help requests'

这是正常的，因为在此配置文件"https://www.khanacademy.org/profile/Kkasparas/"中没有project help requests的值（也没有project help replies）。

因此data['project help requests']和data['project help replies']不存在，因此无法写在csv文件中。

我的目标是使用许多配置文件页面运行此脚本。因此，我想知道在每种情况下都不会放置NA的情况，我不会获取每个变量的数据。然后将NA打印到csv文件中。

换句话说：我想使我的脚本适用于任何类型的用户个人资料页面。

在此先感谢您的贡献：）

Answer 1

在将其写入文件之前，您可以定义一个具有所有可能标题的新列表，并将不存在的键的值设置为“ NA”。

full_data_keys=['questions','votes','answers','flags raised','project help requests','project help replies','comments','tips and thanks']
for header_value in full_data_keys:
    if header_value not in data.keys():
        data[header_value]='NA'

也谨提醒您在问题中提供完整的代码。问题中未定义user_socio_table。我必须查询您以前的问题才能得到。

完整代码为

from bs4 import BeautifulSoup
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.khanacademy.org/profile/Kkasparas/')
r.html.render(sleep=5)
soup=BeautifulSoup(r.html.html,'html.parser')
user_info_table=soup.find('table', class_='user-statistics-table')
dates,points,videos=[tr.find_all('td')[1].text for tr in user_info_table.find_all('tr')]
data = {}
user_socio_table=soup.find_all('div', class_='discussion-stat')
for gettext in user_socio_table:
   category = gettext.find('span')
   category_text = category.text.strip()
   number = category.previousSibling.strip()
   data[category_text] = number
full_data_keys=['questions','votes','answers','flags raised','project help requests','project help replies','comments','tips and thanks']
for header_value in full_data_keys:
    if header_value not in data.keys():
        data[header_value]='NA'
filename = "khanscraptry1.csv"
f = open(filename, "w")
headers = "date, points, videos, questions, votes, answers, flags, project_request, project_replies, comments, tips_thx\n"
f.write(headers)
f.write(dates + "," + points.replace("," , "") + "," + videos + "," + data['questions'] + "," + data['votes'] + "," + data['answers'] + "," + data['flags raised'] + "," + data['project help requests'] + "," + data['project help replies'] + "," + data['comments'] + "," + data['tips and thanks'] + "\n")
f.close()

输出-khanscraptry1.csv

date, points, videos, questions, votes, answers, flags, project_request, project_replies, comments, tips_thx
6 years ago,1527829,1123,25,100,2,0,NA,NA,0,0

如果不存在user_info_table，请更改为以下行

if user_info_table is not None:
    dates,points,videos=[tr.find_all('td')[1].text for tr in user_info_table.find_all('tr')]
else:
    dates=points=videos='NA'

如果在beautifulsoup中没有数据，如何要求f.write（）放入NA？

1 个答案: