读取给定文本文件中的所有单词,并为每个单词打印计数

时间:2018-07-31 11:36:16

标签: python list sorting dictionary

Test.txt包含以下句子(如果土拨鼠可以 卡盘木。)

该程序应该从给定的文本文件中读取所有单词(直到eof) 并打印出每个单词的计数。这个词应该是 处理后的大小写不敏感(全部大写),标点符号应为 删除,输出应按以下顺序排序 频率。

但是我遇到了一个简单的问题,那就是计数行而不是单词,请帮忙一个兄弟。

  

制作翻译表以消除非单词字符

dropChars = "!@#$%ˆ& ()_+-={}[]|\\:;\"’<>,.?/1234567890"
dropDict = dict([(c, '') for c in dropChars])
dropTable = str.maketrans(dropDict)
  

读取文件并建立表。

f = open("Test.txt")
testList=list()
lineNum = 0
table = {} # dictionary: words -> set of line numbers
for line in f:
    testList.append(line)
for line in testList :
    lineNum += 1
    words = line.upper().translate(dropTable).split()
    for word in words:
        if word in table:
            table[word].add(lineNum)
        else:
            table[word] = {lineNum}
f.close()
  

打印表格

for word in sorted(table.keys()):
    print(word, end = ": ")
    for lineNum in sorted(table[word]):
        print(lineNum, end = " ")
    print()

4 个答案:

答案 0 :(得分:1)

首先,您必须确定单词的定义。

定义1:单词是由空格分隔的字符序列。因此,“ you've”是一个单词,“ o'clock”也是一个单词。

定义2:一个单词是“语音或书写的一个独特的有意义的元素”。在这种情况下,“ you've”是两个不同的单词(you + have),而“ o'clock”是一个单独的单词。

因此,如果您运行:

import string
import re
import nltk
import pandas as pd

s = "How much wood would a woodchuck chuck if a woodchuck could chuck wood. \n And also another line you've read from the file with something else. I wake up daily before eight o'clock."

def tokenize(text,semantic=True,sep=" "):
    if semantic:
        #Definition 2
        return nltk.word_tokenize(text)
    else:
        #Definition 1
        return [x for x in text.split(sep) ]

def remove_punctuation(text):
    pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
    return list(filter(None, [pattern.sub('',token) for token in text]))

def lowercase(text):
    return [token.lower() for token in text]

result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(s)))).most_common()

table = pd.DataFrame(result)

table.to_csv('result.csv')

然后您将获得以下csv文件:

enter image description here

请注意,“ ve”(来自“ you've”)是一个独立的单词。

但是,如果在标记化中将语义= True更改为语义= False,

result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(s,semantic=False)))).most_common()

那么您会得到:

enter image description here

但是,在我们的频率表中将“具有”写为“具有”并不是很友好。我们可以通过使用收缩图来解决这个问题。

import string
import re
import nltk
import pandas as pd

s = "How much wood would a woodchuck chuck if a woodchuck could chuck wood. \n And also another line you've read from the file with something else. I wake up daily before eight o'clock."

CONTRACTION_MAP = {"ain't": "is not", "aren't": "are not","can't": "cannot", 
                   "can't've": "cannot have", "'cause": "because", "could've": "could have", 
                   "couldn't": "could not", "couldn't've": "could not have","didn't": "did not", 
                   "doesn't": "does not", "don't": "do not", "hadn't": "had not", 
                   "hadn't've": "had not have", "hasn't": "has not", "haven't": "have not", 
                   "he'd": "he would", "he'd've": "he would have", "he'll": "he will", 
                   "he'll've": "he he will have", "he's": "he is", "how'd": "how did", 
                   "how'd'y": "how do you", "how'll": "how will", "how's": "how is", 
                   "I'd": "I would", "I'd've": "I would have", "I'll": "I will", 
                   "I'll've": "I will have","I'm": "I am", "I've": "I have", 
                   "i'd": "i would", "i'd've": "i would have", "i'll": "i will", 
                   "i'll've": "i will have","i'm": "i am", "i've": "i have", 
                   "isn't": "is not", "it'd": "it would", "it'd've": "it would have", 
                   "it'll": "it will", "it'll've": "it will have","it's": "it is", 
                   "let's": "let us", "ma'am": "madam", "mayn't": "may not", 
                   "might've": "might have","mightn't": "might not","mightn't've": "might not have", 
                   "must've": "must have", "mustn't": "must not", "mustn't've": "must not have", 
                   "needn't": "need not", "needn't've": "need not have","o'clock": "of the clock", 
                   "oughtn't": "ought not", "oughtn't've": "ought not have", "shan't": "shall not",
                   "sha'n't": "shall not", "shan't've": "shall not have", "she'd": "she would", 
                   "she'd've": "she would have", "she'll": "she will", "she'll've": "she will have", 
                   "she's": "she is", "should've": "should have", "shouldn't": "should not", 
                   "shouldn't've": "should not have", "so've": "so have","so's": "so as", 
                   "this's": "this is",
                   "that'd": "that would", "that'd've": "that would have","that's": "that is", 
                   "there'd": "there would", "there'd've": "there would have","there's": "there is", 
                   "they'd": "they would", "they'd've": "they would have", "they'll": "they will", 
                   "they'll've": "they will have", "they're": "they are", "they've": "they have", 
                   "to've": "to have", "wasn't": "was not", "we'd": "we would", 
                   "we'd've": "we would have", "we'll": "we will", "we'll've": "we will have", 
                   "we're": "we are", "we've": "we have", "weren't": "were not", 
                   "what'll": "what will", "what'll've": "what will have", "what're": "what are", 
                   "what's": "what is", "what've": "what have", "when's": "when is", 
                   "when've": "when have", "where'd": "where did", "where's": "where is", 
                   "where've": "where have", "who'll": "who will", "who'll've": "who will have", 
                   "who's": "who is", "who've": "who have", "why's": "why is", 
                   "why've": "why have", "will've": "will have", "won't": "will not", 
                   "won't've": "will not have", "would've": "would have", "wouldn't": "would not", 
                   "wouldn't've": "would not have", "y'all": "you all", "y'all'd": "you all would",
                   "y'all'd've": "you all would have","y'all're": "you all are","y'all've": "you all have",
                   "you'd": "you would", "you'd've": "you would have", "you'll": "you will", 
                   "you'll've": "you will have", "you're": "you are", "you've": "you have" } 

# Credit for this function: https://www.kaggle.com/saxinou/nlp-01-preprocessing-data
def expand_contractions(sentence, contraction_mapping): 

    contractions_pattern = re.compile('({})'.format('|'.join(contraction_mapping.keys())),  
                                      flags=re.IGNORECASE|re.DOTALL) 
    def expand_match(contraction): 
        match = contraction.group(0) 
        first_char = match[0] 
        expanded_contraction = contraction_mapping.get(match) if contraction_mapping.get(match) else contraction_mapping.get(match.lower())                        
        expanded_contraction = first_char+expanded_contraction[1:] 
        return expanded_contraction 

    expanded_sentence = contractions_pattern.sub(expand_match, sentence) 
    return expanded_sentence 

def tokenize(text,semantic=True,sep=" "):
    if semantic:
        #Definition 2
        return nltk.word_tokenize(text)
    else:
        #Definition 1
        return [x for x in text.split(sep) ]

def remove_punctuation(text):
    pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
    return list(filter(None, [pattern.sub('',token) for token in text]))

def lowercase(text):
    return [token.lower() for token in text]

result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(expand_contractions(s,CONTRACTION_MAP))))).most_common()

table = pd.DataFrame(result)

table.to_csv('result.csv')

然后问题解决了。

enter image description here

答案 1 :(得分:0)

f =打开('Test.txt')

cnt = 0 对于f.read()。split()中的单词:     打印(字)     中位数+ = 1     打印cnt

这可能对您有帮助...虽然我也是python的新手。

答案 2 :(得分:0)

此代码:

from collections import Counter
data = open( 'Test1.txt' ).read()  # read the file
data = ''.join( [i.upper() if i.isalpha() else ' ' for i in data] )   # remove the punctuation
c = Counter( data.split() )   # count the words
c.most_common()

打印:

[('A', 2), ('CHUCK', 2), ('WOODCHUCK', 2), ('WOOD', 2), ('WOULD', 1), ('COULD', 1), ('HOW', 1), ('MUCH', 1), ('IF', 1)]

我想知道代码是否太短? =)

答案 3 :(得分:0)

核心python解决方案可能会有所帮助

    data = "How much wood would a woodchuck chuck if a woodchuck could chuck wood."
    data = "".join(i.strip('\n') for i in data if ord(i) < 127)
    data_arr = data.upper().split(' ')
    a = {}
    for i in data_arr:
        if i not in a:
            a[i] = 1
        else:
            a[i] = a[i] + 1
    data = sorted(a.items(), key=lambda a: a[0])
    print(data)