Question

    import sys
    import numpy as np
    import scipy.io as sio
    import matplotlib.pyplot as plt
    from sklearn.cluster import KMeans
    from sklearn.svm import SVC

    filename = sys.argv[1]
    datafile = sio.loadmat(filename)
    data = datafile['bow']
    sizedata=[len(data), len(data[0])]
    gap=[]
    SD=[]    
    for knum in xrange(10,20):
            print knum
            #Clustering original Data
            kmeanspp = KMeans(n_clusters=knum,init = 'k-means++',max_iter = 100,n_jobs = 1)
            kmeanspp.fit(data)
            dispersion = kmeanspp.inertia_
            #Clustering Reference Data
            nrefs = 10
            refDisp = np.zeros(nrefs)
            for nref in xrange(nrefs):
                refdata = np.random.random_sample((sizedata[0],sizedata[1]))
                refkmeans = KMeans(n_clusters=knum,init='k-means++',max_iter=100,n_jobs=1)
                refkmeans.fit(refdata)
                refdisp = refkmeans.inertia_
                refDisp[nref]=np.log(refdisp)
            mean_log_refdisp = np.mean(refDisp)
            gap.append(mean_log_refdisp-np.log(dispersion))
            #Calculating standard deviaiton
            sd = (sum([(r-m)**2 for r,m in zip(refDisp,[mean_log_refdisp]*nrefs)])/nrefs)**0.5
            SD.append(sd)
    SD = [sd*((1+(1/nrefs))**0.5) for sd in SD]
    #determining optimal k
    opt_k = None
    diff = []
    for i in xrange(len(gap)-1):
        diff = (SD[i+1]-(gap[i+1]-gap[i]))
        if diff>0:
            opt_k = i+10
            break
    print diff
    plt.plot(np.linspace(10,19,10,True),gap)
    plt.show()

Here I am trying to implement the Gap Statistic method for determining the optimal number of clusters. But the problem is that every time I run the code I get a different value for k. What is the solution to the problem? How can the value of optimal k differ for the same data?

I have stored the data in a .mat file beforehand and I am passing it as an argument via terminal

I am looking for the smallest value of k for which Gap(k)>= Gap(k+1)-s(k+1) where s(k+1) = sd(k+1)*square_root(1+(1/B)) where sd is the standard deviation of the reference distribution and B is the number of copies of Monte Carlo sample

Otherwise stated, I am searching for the value of k for which

s(k+1)-Gap(k+1)+Gap(k)>=0

Answer 1

模拟问题：

1- sd = (sum([(r-m)**2 for r,m in zip(refDisp,[mean_log_refdisp]*nrefs)])/nrefs)**0.5

为什么您要根据原始论文将zip的第二个部分乘以不需要的nrefs。

2-

 if diff>0:
        opt_k = i+10
        break

如果diff> 0，则您希望diff> = 0，因为可能发生相等关于为什么每次获得不同数目的聚类的原因，正如人们所说的那样，这是蒙特卡洛模拟，因此可能存在随机性，而且还取决于您正在聚类的内容和数据集。我建议您针对Silhouette和Elbow测试算法，以更好地了解簇数。

Answer 2

一个选项是多次运行你的函数，然后平均差距统计和s值，找到平均s（k + 1）-Gap（k + 1）+ Gap（k）的最小k大于

这将花费更长时间，但会给出更可靠的结果。

Gap Statistic Method

2 个答案: