我正在使用Python s scikit-learn lib来学习集群，但是我找不到找到最佳集群数的方法。我试图列出群集数量并在for loop中传递它，并查看elbow，但是我想找到更好的解决方案。仅当该行变得非常平滑并且我看不到range(1,11)之后，我才对elbow执行此方法。
我尝试过silhouette_score，但得到的值很低，有时是负数。
此外，我使用文本数据，我写了一些可以*（可以说）分组的句子，我有关于房屋/房屋，学习，聚会，食物的句子...
我是否有可能因为我使用文本数据而使silhouette_score的值变低，我还需要在cv.fit_transform(doc)之后缩放数据吗？
是否有更好的方法，也许某些函数可以返回最佳簇数的integer值？例如1,2,3,4 .... n
这是我编写的代码：
import sklearn.metrics as sm
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.preprocessing import scale
from sklearn.cluster import KMeans, SpectralClustering, MiniBatchKMeans
from sklearn.metrics import silhouette_samples, silhouette_score
import matplotlib.pyplot as plt
doc = ['this is very good show' , 'i had a great time on my school trip', 'such a boring movie', 'Springbreak was amazing', 'You are wrong', 'let s go to the beach', 'how can we do this',
'i love this product', 'this is an amazing item', 'this food is delicious', 'I had a great time last night', 'thats right', ' lets go to the party', 'we were at the party last night',
'this is my favourite restaurant, I love their food, its so good','i love healty food', 'skiing is the best sport', 'what is this', 'this product has a lot of bugs', "i'm on the road again",
'I love basketball, its very dynamic' , 'its a shame that you missed the trip, it was amazing', 'Party last night was so boring', 'lets go on road trip', 'this is my home, im living there for 26 years',
'such a nice song' , 'this is the best movie ever', 'hawaii is the best place for trip','how that happened','This is my favourite band', 'true love', 'party was great','home sweet home',
'I cant believe that you did that', 'Why are you doing that, I do not get it', 'this is tasty', 'this song is amazing', 'this food is tasty', 'lets go to the cinema', 'lets get together at my house',
'I need to study for the test', 'I cant go out this weekend', 'I had a great time last night', 'I went out last night and it was amazing', 'you are beautiful', 'we crashed the party',
'this is the best song i have ever heard', 'i love listening to music', 'music is my life', 'this song is terrible', 'how was your hollyday', 'i do not understand you, I have told you that last night',
'I know whats best for you', 'I m on collage now', 'this is my favourite subject', 'math is fun', 'i love to study maths', 'programming is my live', 'i need to study, my final exam is tomorrow',
'i m cooming home', 'i need to clean my house', 'what do you thing about last night', 'lets go out, my house is a mess', 'Im staying at home tonight', 'love is such a beautiful word',
'i want to buy new house for me and my family', 'im will be home in a couple of hours', 'im working on a science project', 'working is hard and i need to work', 'you need to find a job',
'this is bad, and we cant do anything about that', 'real estate market is growing', 'im selling my appartment', 'i live at the appartment above', 'i m into real estate', 'prices are going down',
'i m building house of cards', 'I feel so tired, i was studying all nigh long', 'i was playing piano for more than 10 years and I was pretty good at it','I have never done that in my life',
'i will buy this product in a couple of days', 'i m buying new phone next month', 'my home is near by', 'i m living in my home', 'i live in my parents house', 'i m living in my appartment',
'my phone is very slow', 'do you know password for wifi', 'wifi is short for wireless network', 'you are so funny', 'my neighbours are horrible', 'such a nice phone, im glad to have it',
'last time we went into that club and it was so boring', 'if I were you, i would never said that', 'you done very good work, your boss is very proud of you', 'Overall, I like this place a lot',
'I was spending money on wrong things', 'whats the price for this item', 'where can I buy it', 'is it for sale', 'This hole in the wall has great Mexican street tacos, and friendly staff'
'The movie showed a lot of Florida at it s best, made it look very appealing', 'This short film certainly pulls no punches', 'This is the kind of money that is wasted properly',
'Not only did it only confirm that the film would be unfunny and generic, but it also managed to give away the ENTIRE movie', 'But it s just not funny','you have already done that',
'I especially liked the non-cliche choices with the parents', 'it was well-paced and suited its relatively short run time']
cv = TfidfVectorizer(analyzer = 'word', max_features = 4000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')
x = cv.fit_transform(doc)
my_list = []
for i in range(1,10):
kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 0)
kmeans.fit(x)
my_list.append(kmeans.inertia_)
silhouette_avg = silhouette_score(x, cluster_labels)
print(silhouette_avg)
plt.plot(range(1,10),my_list)
plt.show()
答案 0 :(得分：0)
通常，查找最佳簇数是一个难题，因为它没有唯一的解决方案，并且该问题也不是确定性的（尤其是对于文本数据）。此外，聚类问题的最佳解决方案是在您使用的模型后面优化给定度量的局部最优，并且存在大量clustering models。
因此，自动学习文本数据簇的“正确”数目的基准就是所谓的分层Dirichlet流程（HDP），它概括了潜在的Dirichlet分配（LDA）模型。
您可以在gensim library中找到HDP的示例和用例。