Question

I have extracted the list of sentences from a document. I am pre-processing this list of sentences to make it more sensible. I am faced with the following problem

I have sentences such as Java is a prog rammng lan guage. C is a gen eral purpose la nguage.

I would like to correct such sentences using a look up dictionary? to remove the unwanted spaces.

The final output should be Java is a programmng language. C is a general purpose language.

I need help with some pointers to look for such approaches. How to solve the above problem?

I want to solve the above problem using python code. Thanks.

Answer 1

如果您想正确拼写和解析单词，则需要进行拼写检查。这是一个拼写检查器，用于导入“ re”名称空间以及完整的文章here ...

import re
from collections import Counter

def words(text): return re.findall(r'\w+', text.lower())

WORDS = Counter(words(open('big.txt').read()))

def P(word, N=sum(WORDS.values())): 
    "Probability of `word`."
    return WORDS[word] / N

def correction(word): 
    "Most probable spelling correction for `word`."
    return max(candidates(word), key=P)

def candidates(word): 
    "Generate possible spelling corrections for `word`."
    return (known([word]) or known(edits1(word)) or known(edits2(word)) or [word])

def known(words): 
    "The subset of `words` that appear in the dictionary of WORDS."
    return set(w for w in words if w in WORDS)

def edits1(word):
    "All edits that are one edit away from `word`."
    letters    = 'abcdefghijklmnopqrstuvwxyz'
    splits     = [(word[:i], word[i:])    for i in range(len(word) + 1)]
    deletes    = [L + R[1:]               for L, R in splits if R]
    transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]
    replaces   = [L + c + R[1:]           for L, R in splits if R for c in letters]
    inserts    = [L + c + R               for L, R in splits for c in letters]
    return set(deletes + transposes + replaces + inserts)

def edits2(word): 
    "All edits that are two edits away from `word`."
    return (e2 for e1 in edits1(word) for e2 in edits1(e1))`

它不仅可以修复拆分的单词，还可以删除，转置和插入不规则单词以“更正”它们。您可以将“ big.txt”文件替换为Counter构造函数中正在使用的文档，希望一切都可以从那里开始。

Answer 2

这是一个适用于您的示例的简单脚本。显然，您想要更大的有效单词语料库。另外，如果加入下一个单词后无法修复非单词，您可能希望拥有一个elif分支来回头一个单词。

from string import punctuation

word_list = "big list of words including a programming language is general purpose"
valid_words = set(word_list.split())

bad = "Java is a prog ramming lan guage. C is a gen eral purpose la nguage."
words = bad.split()

out_words = []
i = 0
while i < len(words):
    word = words[i]
    if word not in valid_words and i+1 < len(words):
        next_word = words[i+1]
        joined = word + next_word
        if joined.strip(punctuation) in valid_words:
            word = joined
            i += 1
    out_words.append(word)
    i += 1

good = " ".join(out_words)
print(good)

Fix the words and remove the unwanted spaces between splitted word using python?

2 个答案: