如何计算从文件读取的字符串中的单词？

Question

我正在尝试创建一个程序，该程序将所有文本文件存储在给定路径中，并将所有字符串保存在一个列表中：

import os
import collections

vocab = set()
path = 'a\\path\\'

listing = os.listdir(path)
unwanted_chars = ".,-_/()*"
vocab={}
for file in listing:
    #print('Current file : ', file)
    pos_review = open(path+file, "r", encoding ='utf8')
    words = pos_review.read().split()
    #print(type(words))
    vocab.update(words)
pos_review.close()

print(vocab)
pos_dict = dict.fromkeys(vocab,0)
print(pos_dict)

输入

file1.txt: A quick brown fox.
file2.txt: a quick boy ran.
file3.txt: fox ran away.

输出

A : 2
quick : 2
brown : 1
fox : 2
boy : 1
ran : 2
away : 1

直到现在，我都可以为这些字符串制作字典。但是现在不确定如何在所有文本文件中组合键，值对和它们的频率。

Answer 1

希望这会有所帮助，

import os
import collections

vocab = set()
path = 'a\\path\\'

listing = os.listdir(path)
unwanted_chars = ".,-_/()*"
vocab={}
whole=[]
for file in listing:
    #print('Current file : ', file)
    pos_review = open(path+file, "r", encoding ='utf8')
    words = pos_review.read().split()
    whole.extend(words)
pos_review.close()

print(vocab)

d={} #Creating an Empty dictionary
for item in whole:
    if item in d.keys():
        d[item]+=1 #Update count
    else:
        d[item]=1
print(d)

Answer 2

使用collections.Counter：

• Counter是用于计数可迭代项的dict子类

数据

• 给出3个文件，分别名为t1.txt，t2.txt和t3.txt
• 每个文件包含以下3行文本

file1 txt A quick brown fox.
file2 txt a quick boy ran.
file3 txt fox ran away.

代码：

获取文件：

• pathlib

from pathlib import Path

files = list(Path('e:/PythonProjects/stack_overflow/t-files').glob('t*.txt'))
print(files)

# Output
[WindowsPath('e:/PythonProjects/stack_overflow/t-files/t1.txt'),
 WindowsPath('e:/PythonProjects/stack_overflow/t-files/t2.txt'),
 WindowsPath('e:/PythonProjects/stack_overflow/t-files/t3.txt')]

收集并计算单词：

• 创建一个单独的函数clean_str，以清除每一行文本
• str.lower代表小写字母
• str.translate，str.maketrans和string.punctuation用于高度优化的标点删除
  • 来自Best way to strip punctuation from a string

from collections import Counter
import string

def clean_string(value: str) -> list:
    value = value.lower()
    value = value.translate(str.maketrans('', '', string.punctuation))
    value = value.split()
    return value

words = Counter()
for file in files:
    with file.open('r') as f:
        lines = f.readlines()
        for line in lines:
            line = clean_string(line)
            words.update(line)

print(words)

# Output
Counter({'file1': 3,
         'txt': 9,
         'a': 6,
         'quick': 6,
         'brown': 3,
         'fox': 6,
         'file2': 3,
         'boy': 3,
         'ran': 6,
         'file3': 3,
         'away': 3})

words的列表：

list_words = list(words.keys())
print(list_words)

>>> ['file1', 'txt', 'a', 'quick', 'brown', 'fox', 'file2', 'boy', 'ran', 'file3', 'away']

Answer 3

这也有效

import pandas as pd
import glob.glob

files = glob.glob('test*.txt')
txts = []

for f in files:
    with open (f,'r') as t: txt = t.read()
    txts.append(txt)

texts=' '.join(txts)
df = pd.DataFrame({'words':texts.split()})
out = df.words.value_counts().to_dict()