Question

我想计算JSON文件中连续文章对的余弦相似度。到目前为止，我设法做到了但是......我只是意识到，在转换每篇文章的tfidf时，我没有使用文件中所有文章中的术语，而只使用每对中的条款。这是我正在使用的代码，它提供每个连续文章对的余弦相似系数。

import json
import nltk
with open('SDM_2015.json') as f:
    data = [json.loads(line) for line in f]

## Loading the packages needed:
import nltk, string
from sklearn.feature_extraction.text import TfidfVectorizer

## Defining our functions to filter the data

# Short for stemming each word (common root)
stemmer = nltk.stem.porter.PorterStemmer()

# Short for removing puctuations etc
remove_punctuation_map = dict((ord(char), None) for char in string.punctuation)

## First function that creates the tokens
def stem_tokens(tokens):
    return [stemmer.stem(item) for item in tokens]

## Function that incorporating the first function, converts all words into lower letters and removes puctuations maps (previously specified)
def normalize(text):
    return stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))

## Lastly, a super function is created that contains all the previous ones plus stopwords removal
vectorizer = TfidfVectorizer(tokenizer=normalize, stop_words='english')

## Calculation one by one of the cosine similatrity

def foo(x, y):
    tfidf = vectorizer.fit_transform([x, y])
    return ((tfidf * tfidf.T).A)[0,1]

my_funcs = {}
for i in range(len(data) - 1):
    x = data[i]['body']
    y = data[i+1]['body']
    foo.func_name = "cosine_sim%d" % i
    my_funcs["cosine_sim%d" % i] = foo
    print(foo(x,y))

如何使用JSON文件中可用的所有文章的整个术语来开发余弦相似性，而不仅仅是每对文件中的那些？

亲切的问候，

安德烈

Answer 1

我认为，基于我们上面的讨论，您需要更改foo函数以及下面的所有内容。请参阅下面的代码。请注意，我实际上并没有运行它，因为我没有您的数据，也没有提供样本行。

## Loading the packages needed:
import nltk, string
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import cosine
import json
from  sklearn.metrics.pairwise import cosine_similarity

with open('SDM_2015.json') as f:
    data = [json.loads(line) for line in f]

## Defining our functions to filter the data

# Short for stemming each word (common root)
stemmer = nltk.stem.porter.PorterStemmer()

# Short for removing puctuations etc
remove_punctuation_map = dict((ord(char), None) for char in string.punctuation)

## First function that creates the tokens
def stem_tokens(tokens):
    return [stemmer.stem(item) for item in tokens]

## Function that incorporating the first function, converts all words into lower letters and removes puctuations maps (previously specified)
def normalize(text):
    return stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))

## tfidf
vectorizer = TfidfVectorizer(tokenizer=normalize, stop_words='english')
tfidf_data = vectorizer.fit_transform(data)

#cosine dists
similarity matrix  = cosine_similarity(tfidf_data)

JSON文件

1 个答案: