Question

我试图将安然电子邮件的所有主体附加到一个文件中，这样我就可以通过消除停用词并用NLTK将其拆分成句子来处理这些电子邮件的文本。我的问题是转发和回复的消息，我不知道如何清理它们。到目前为止，这是我的代码：

    import os, email, sys, re,nltk, pprint 
    from email.parser import Parser

    rootdir = '/Users/art/Desktop/maildir/lay-k/elizabeth'
    #function that appends all the body parts of Emails
    def email_analyse(inputfile,  email_body):
        with open(inputfile, "r") as f:
        data = f.read()

        email = Parser().parsestr(data)

        email_body.append(email.get_payload())
    #end of function
    #defining a list that will contain bodies
    email_body = []
    #call the function email_analyse for every function in directory
    for directory, subdirectory, filenames in  os.walk(rootdir):
        for filename in filenames:
            email_analyse(os.path.join(directory, filename),  email_body )
    #the stage where I clean the emails

    with open("email_body.txt", "w") as f:
        for val in email_body:
            if(val):
                val = val.replace("\n", "")
                val = val.replace("=01", "")
                #for some reason I had many of ==20 and =01 in my text
                val = val.replace("==20", "")
                f.write(val)
                f.write("\n")

这是部分输出： Well, with the photographer and the band, I would say we've pretty much outdone our budget! Here's the information on the photographer. I have a feeling for some of the major packages we could negotiate at least a couple of hours at the rehearsal dinner. I have no idea how much this normally costs, but he isn't cheap!---------------------- Forwarded by Elizabeth Lay/HOU/AZURIX on 09/13/99 07:34 PM ---------------------------acollins@reggienet.com on 09/13/99 05:37:37 PMPlease respond to acollins@reggienet.com To: Elizabeth Lay/HOU/AZURIX@AZURIXcc: Subject: Denis Reggie Wedding PhotographyHello Elizabeth:Congratulations on your upcoming marriage! I am Ashley Collins, Mr.Reggie's Coordinator. Linda Kessler forwarded your e.mail address to me sothat I may provide you with information on photography coverage for Mr.Reggie's wedding photography. 所以结果不是纯文本。关于如何做到这一点的任何想法？

Answer 1

您可能希望查看正则表达式来解析转发和回复文本，因为格式应该在整个语料库中保持一致。

要删除转发的文本，您可以使用以下正则表达式：

-{4,}(.*)(\d{2}:\d{2}:\d{2})\s*(PM|AM)

将匹配四个或更多连字符之间的所有内容以及格式XX：XX：XX PM中的时间。匹配3个破折号也可能正常。我们只是想避免在电子邮件正文中匹配连字符和em-dashes。您可以使用此正则表达式并编写自己的匹配To和Subject标题：https://regex101.com/r/VGG4bu/1/

您还可以查看NLTK一书中的3.4节，该书讨论Python中的正则表达式：http://www.nltk.org/book/ch03.html

祝你好运！这听起来像一个有趣的项目。

Answer 2

如果您仍然对此问题感兴趣，那么我将为Enron数据集专门创建一个预处理脚本。您会注意到，一封新电子邮件将始终以标签“主题：”开头，我实现了一个功能，该功能删除该标签左侧的所有文本，仅删除最后一个“主题：”标签上的所有文本，以删除所有转发的邮件。具体代码：

# Cleaning content column
df['content'] = df['content'].str.rsplit('Subject: ').str[-1] 
df['content'] = df['content'].str.rsplit(' --------------------------- ').str[-1]

总体脚本（如果有兴趣的话）：

# Importing the dataset, and defining columns
import pandas as pd
df = pd.read_csv('enron_05_17_2015_with_labels_v2.csv', usecols=[2,3,4,13], dtype={13:str})

# Building a count of how many people are included in an email
df['Included_In_Email'] = df.To.str.count(',')
df['Included_In_Email'] = df['Included_In_Email'].apply(lambda x: x+1)

# Dropping any NaN's, and emails with >15 recipients
df = df.dropna()
df = df[~(df['Included_In_Email'] >=15)]

# Seperating remaining emails into a line-per-line format
df['To'] = df.To.str.split(',')
df2 = df.set_index(['From', 'Date', 'content', 'Included_In_Email']) 
['To'].apply(pd.Series).stack()
df2 = df2.reset_index()
df2.columns = ['From','To','Date','content', 'Included_In_Email']

# Renaming the new column, dropping unneeded column, and changing indices
del df2['level_4']
df2 = df2.rename(columns = {0: 'To'})
df2 = df2[['Date','From','To','content','Included_In_Email']]
del df

# Cleaning email addresses
df2['From'] = df2['From'].map(lambda x: x.lstrip("frozenset"))
df2['To'] = df2['To'].map(lambda x: x.lstrip("frozenset"))
df2['From'] = df2['From'].str.strip("<\>(/){?}[:]*, ")
df2['To'] = df2['To'].str.strip("<\>(/){?}[:]*, ")
df2['From'] = df2['From'].str.replace("'", "")
df2['To'] = df2['To'].str.replace("'", "")
df2['From'] = df2['From'].str.replace('"', "")
df2['To'] = df2['To'].str.replace('"', "")

# Acccounting for users having different emails
email_dict = pd.read_csv('dict_email.csv')    
df2['From'] = df2.From.replace(email_dict.set_index('Old')['New'])
df2['To'] = df2.To.replace(email_dict.set_index('Old')['New'])
del email_dict

# Removing emails not containing @enron
df2['Enron'] = df2.From.str.count('@enron')
df2['Enron'] = df2['Enron']+df2.To.str.count('@enron')
df2 = df2[df2.Enron != 0]
df2 = df2[df2.Enron != 1]
del df2['Enron']

# Adding job roles which correspond to staff
import csv
with open('dict_role.csv') as f:
   role_dict = dict(filter(None, csv.reader(f)))
df2['Sender_Role'] = df2['From'].map(role_dict)
df2['Receiver_Role'] = df2['To'].map(role_dict)
df2 = df2[['Date','From','To','Sender_Role','Receiver_Role','content','Included_In_Email']]
del role_dict

# Cleaning content column
df2['content'] = df2['content'].str.rsplit('Subject: ').str[-1] 
df2['content'] = df2['content'].str.rsplit(' --------------------------- ').str[-1] 

# Condensing records into one line per email exchange, adding weights
Weighted = df2.groupby(['From', 'To']).count()

# Adding weight column, removing redundant columns, splitting indexed column
Weighted['Weight'] = Weighted['Date']
Weighted = 
Weighted.drop(['Date','Sender_Role','Receiver_Role','content','Included_In_Email'], 1)
Weighted.reset_index(inplace=True)

# Re-adding job-roles to staff
with open('dict_role.csv') as f:
   role_dict = dict(filter(None, csv.reader(f)))
Weighted['Sender_Role'] = Weighted['From'].map(role_dict)
del role_dict

# Dropping exchanges with a weight of <= x, or no identifiable role
Weighted2 = Weighted[~(Weighted['Weight'] <=3)]
Weighted2 = Weighted.dropna()

脚本中使用了两个字典（用于匹配工作角色并更改同一个人的多封电子邮件），可以在here中找到。

如何从安然电子邮件正文中删除“转发邮件”标题和不需要的内容？

2 个答案: