Question

我有一个包含其他文件夹的文件夹，每个文件夹都包含大量文本文件，大约32214个文件。我想在一个特定单词之前和之后打印5个单词，我的代码应该读取所有这些文件。下面的代码可以工作但是大约需要8个小时来阅读所有文件并提取句子。如何更改代码，以便在几分钟内读取和打印句子？（语言是波斯语）

.
.
.
def extact_sentence ():
    f= open ("پاکت", "w", encoding = "utf-8")
    y = "پاکت"
    text= normal_text(folder_path) # the first function to normalize the files
    for i in text:
        for line in i:
            split_line = line.split()
            if y in split_line:
                index = split_line.index(y)
                d = (' '.join(split_line[max(0,index-5):min(index+6,len(split_line))]))
                f.write(d + "\n")
    f.close()

enter image description here

Answer 1

使用os.walk访问所有文件。然后在每个文件上使用滚动窗口，并检查每个窗口的中间字：

import os


def getRollingWindow(seq, w):
    win = [next(seq) for _ in range(window_size)]
    yield win
    for e in seq:
        win[:-1] = win[1:]
        win[-1] = e
        yield win


def extractSentences(rootDir, searchWord):
    with open("پاکت", "w", encoding="utf-8") as outfile:
        for root, _dirs, fnames in os.walk(rootDir):
            for fname in fnames:
                print("Looking in", os.path.join(root, fname))
                with open(os.path.join(root, fname)) as infile:
                    for window in getRollingWindow(word for line in infile for word in line.split(), 11):
                        if window[5] != searchWord: continue
                        outfile.write(' '.join(window))

在文本文件中的特定单词之前和之后获取单词

1 个答案: