Question

我想从我提到的例句中得到bigrams和trigrams。

我的代码适用于bigrams。但是，它没有捕获数据中的三元组（例如，我的句子的5个地方提到的人机交互）

方法1 下面提到的是我在Gensim中使用短语的代码。

from gensim.models import Phrases
documents = ["the mayor of new york was there", "human computer interaction and machine learning has now become a trending research area","human computer interaction is interesting","human computer interaction is a pretty interesting subject", "human computer interaction is a great and new subject", "machine learning can be useful sometimes","new york mayor was present", "I love machine learning because it is a new subject area", "human computer interaction helps people to get user friendly applications"]
sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream, min_count=1, threshold=1, delimiter=b' ')
trigram = Phrases(bigram_phraser[sentence_stream])

for sent in sentence_stream:
    bigrams_ = bigram_phraser[sent]
    trigrams_ = trigram[bigrams_]

    print(bigrams_)
    print(trigrams_)

方法2 我甚至尝试过使用Phraser和Phrases，但它没有用。

from gensim.models import Phrases
from gensim.models.phrases import Phraser
documents = ["the mayor of new york was there", "human computer interaction and machine learning has now become a trending research area","human computer interaction is interesting","human computer interaction is a pretty interesting subject", "human computer interaction is a great and new subject", "machine learning can be useful sometimes","new york mayor was present", "I love machine learning because it is a new subject area", "human computer interaction helps people to get user friendly applications"]
sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream, min_count=1, threshold=2, delimiter=b' ')
bigram_phraser = Phraser(bigram)
trigram = Phrases(bigram_phraser[sentence_stream])

for sent in sentence_stream:
    bigrams_ = bigram_phraser[sent]
    trigrams_ = trigram[bigrams_]

    print(bigrams_)
    print(trigrams_)

请帮我解决这个三卦问题。

我正在关注Gensim的example documentation。

Answer 1

我能够通过对代码进行一些修改来获取bigrams和trigrams：

from gensim.models import Phrases
documents = ["the mayor of new york was there", "human computer interaction and machine learning has now become a trending research area","human computer interaction is interesting","human computer interaction is a pretty interesting subject", "human computer interaction is a great and new subject", "machine learning can be useful sometimes","new york mayor was present", "I love machine learning because it is a new subject area", "human computer interaction helps people to get user friendly applications"]
sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream, min_count=1, delimiter=b' ')
trigram = Phrases(bigram[sentence_stream], min_count=1, delimiter=b' ')

for sent in sentence_stream:
    bigrams_ = [b for b in bigram[sent] if b.count(' ') == 1]
    trigrams_ = [t for t in trigram[bigram[sent]] if t.count(' ') == 2]

    print(bigrams_)
    print(trigrams_)

我从bigram threshold = 1中删除了Phrases参数，因为否则它似乎形成了奇怪的数字，允许构造奇怪的三元组（注意bigram用于构建三元组{ {1}}）;当您有更多数据时，此参数可能会有用。对于trigrams，还需要指定Phrases参数，因为如果未提供，则默认为5.

为了检索每个文档的双字母组和三元组，您可以使用此列表理解技巧来过滤分别由两个或三个单词组成的元素。

修改 - 有关min_count参数的一些详细信息：

估算工具使用此参数来确定两个单词 a 和 b 是否构成短语，并且仅在以下情况下：

threshold

其中 N 是总词汇量大小。默认情况下，参数值为10（请参阅docs）。因此，(count(a followed by b) - min_count) * N/(count(a) * count(b)) > threshold越高，单词形成短语的约束就越难。

例如，在您的第一种方法中，您尝试使用threshold，因此您将获得threshold = 1作为5个句子中的3个以2＃人机交互＆＃34开头的数字34 ;;奇怪的第二个数字是更宽松的阈值的结果。

然后，当您尝试使用默认['human computer','interaction is']获取三元组时，您只能获得这些3个句子的threshold = 10，而其余两个句子则没有任何内容（按阈值过滤）;因为这是一个4克而不是三元组，它也会被['human computer interaction is']过滤掉。例如，如果您将三元组阈值降低到1，则可以将[＆＃39;人机交互＆＃39;]作为其余两个句子的三元组。获得更好的参数组合似乎并不容易，here's更多关于它的信息。

我不是专家，所以最后得出这样的结论：我认为最好先获得好的数字结果（而不是像＃39;互动是＆＃39;）之前继续前进，因为奇怪的数字会给进一步的三卦增加混乱，4克......

使用Gensim获取三元组的问题

1 个答案: