Question

我尝试使用python实现spark的kmean算法，我使用位于库http://spark.apache.org/docs/latest/mllib-clustering.html#k-means中的python代码代码正在运行，但为了制作我需要操作对象的情节＆＃34; sameModel = KMeansModel.load（sc，＆＃34; KMeansModel2＆＃34;）＆＃34;，我不知道知道该怎么做..我应该在csv文件中加载sc吗？请帮助！

  from __future__ import print_function
  # $example on$
  from numpy import array
  from math import sqrt
  # $example off$
  from pyspark import SparkContext
  # $example on$
  from pyspark.mllib.clustering import KMeans, KMeansModel
  # $example off$
  if __name__ == "__main__":
    sc = SparkContext(appName="KMeansExample")  # SparkContext
    # $example on$
    # Load and parse the data
    data = sc.textFile("kmeans_data2.txt")
    parsedData = data.map(lambda line: array([float(x) for x in      line.split(' ')]))
    # Build the model (cluster the data)
    clusters = KMeans.train(parsedData, 2, maxIterations=10,    initializationMode="random")
    # Evaluate clustering by computing Within Set Sum of Squared Errors
    def error(point):
        center = clusters.centers[clusters.predict(point)]
        return sqrt(sum([x**2 for x in (point - center)]))
WSSSE = parsedData.map(lambda point: error(point)).reduce(lambda x,y:x + y)
    print("Within Set Sum of Squared Error = " + str(WSSSE))
    # Save and load model
    clusters.save(sc, "KMeansModel2")
    sameModel = KMeansModel.load(sc, "KMeansModel2")
    print (sameModel)
    # $example off$
    sc.stop()

Answer 1

如果您想使用模式（预测），您需要在正在运行的火花环境中执行此操作（例如，您可以set it up in a jupyter notebook）

K表示lib spark python

1 个答案: