python:基于字典

时间:2017-09-20 16:10:11

标签: python nlp

我有词典

dict = ["as", "ass", "share", "rest"]

和字符串输入

string = "xassharest"

我希望根据这样的字典显示所有可能的单词:

[('x', 'as', 's', 'h', 'a', 'rest'), ('x', 'as', 'share', 's', 't'), ('x', 'ass', 'h', 'a', 'rest')]

实际上我已经尝试过使用字符串的所有组合(使用库itertools),但这需要很长时间。这是我的代码:

def getallpossiblewords(string):
    allwords = preprocessingcorpus("corpus.txt")
    temp = []
    for i in range(0, len(string)):
        for j in range(1, len(string) + 1):
            if string[i:j] in allwords:
                temp += [string[i:j]]

    allposwords = sorted(temp, key=len, reverse=True)
    #print(allposwords)
    return allposwords

def wordseg(string):
    a = string
    b = getallpossiblewords(string)
    cuts = []
    allpos = []
    for i in range(0,len(a)):
        cuts.extend(combinations(range(1,len(a)),i))
    for i in cuts:
        last = 0
        output = []
        for j in i:
            output.append(a[last:j])
            last = j
        output.append(a[last:])
        for x in range(len(output)):
            if output[x] in b:
                allpos += [output]
                #print(output)
    #print(allpos)

    fixallpos = list()
    for sublist in allpos:
        if sublist not in fixallpos:
            fixallpos.append(sublist)

我需要最快的算法来解决这个问题,因为字符串的输入可能更长。

任何人都可以解决我的问题吗?

1 个答案:

答案 0 :(得分:2)

这似乎是对str.partition()的完美递归使用。下面是我的示例实现,我不会声称解决所有问题(因为几乎没有测试用例),而是试图在这个特定的方法上做销售工作:

def segmented(string):

    segmentations = set()

    for word in words:
        before, match, after = string.partition(word)

        if not match:
            continue

        prefixes = segmented(before) or [before]
        suffixes = segmented(after) or [after]

        if prefixes and suffixes:
            for prefix in prefixes:
                for suffix in suffixes:
                    segmentations.add((*prefix, word, *suffix))
        elif prefixes:
            for prefix in prefixes:
                    segmentations.add((*prefix, word, *suffixes))
        elif suffixes:
            for suffix in suffixes:
                    segmentations.add((*prefixes, word, suffix))
        else:
            segmentations.add((*prefixes, word, *suffixes))

    return segmentations

words = ["as", "ass", "share", "rest"]

print(segmented("xassharest"))

<强>输出

% python3 test.py
{('x', 'as', 's', 'h', 'a', 'rest'), ('x', 'as', 'share', 's', 't'), ('x', 'ass', 'h', 'a', 'rest')}
%