Question

我需要计算给定语料库中单词之间的正交相似度（编辑/ Levenshtein距离）。

正如基里尔在下面所建议的那样，我试图做到以下几点：

import csv, itertools, Levenshtein
import numpy as np

# import the list of words from csv file
path = '/Users/my path'
file = path + 'file.csv'

with open(file, 'rb') as f:
    reader = csv.reader(f)
    wordlist = list(reader)

wordlist = np.array(wordlist) #make it a np array
wordlist2 = wordlist[:,0] #subset the first column of the imported list

for a, b in itertools.product(wordlist, wordlist):
    if a < b:
        print(a, b, Levenshtein.distance(a, b))

但是，会弹出以下错误：

ValueError：具有多个元素的数组的真值是不明确的。使用a.any（）或a.all（）

我理解代码中的歧义，但有人可以帮我弄清楚如何解决这个问题吗？谢谢！

Answer 1

Levenshtein距离的定义只能在两个字符串之间计算：它是如何编辑一个字符串以获得另一个字符串的。您可以成对比较单词，需要n*(n-1)/2次比较（其中n是您语料库中唯一单词的数量）。以下是如何做到的：

>>> import itertools, Levenshtein
>>> words = sorted(set('little Mary had a little lamb'.split()))
>>> for a, b in itertools.product(words, words):
...     if a < b:
...         print(a, b, Levenshtein.distance(a, b))
... 
Mary a 3
Mary had 3
Mary lamb 3
Mary little 6
a had 2
a lamb 3
a little 6
had lamb 3
had little 6
lamb little 5

Answer 2

感谢Kirill的帮助，这是我想出的代码。

import csv#, StringIO
import itertools, Levenshtein

# open the newline-separated list of words
path = '/Users/your path'
file = path + 'wordlists.txt'
output = path + 'ortho_similarities.txt'
words = sorted(set(s.strip() for s in open(file)))

# the following loop take all possible pairwise combinations
# of the words in the list words, and calculate the LD
# and then let's write everything in a csv file
with open(output, 'wb') as f:
   writer = csv.writer(f, delimter=",", lineterminator="\n")
   for a, b in itertools.product(words, words):
      if a < b:
        write.writerow([a, b, Levenshtein.distance(a,b)])

python - 计算列表单词之间的正交相似度

2 个答案: