Question

我正在尝试将我的Kmean群集模型保存到本地文件系统中。我正在使用Pyspark mllib进行Kmean聚类。但是我收到以下错误。

    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o46.save.
: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:E:/Work/Python1/work/spark/anomalydetectionspark/test/spark-warehouse
    at org.apache.hadoop.fs.Path.initialize(Path.java:206)
    at org.apache.hadoop.fs.Path.<init>(Path.java:172)
    at org.apache.hadoop.fs.Path.<init>(Path.java:89)

我的代码：

  clusters = KMeans.train(parsedData, 2, maxIterations=10,
                            runs=10, initializationMode="random")

    # Evaluate clustering by computing Within Set Sum of Squared Errors
    def error(point):
        center = clusters.centers[clusters.predict(point)]
        return sqrt(sum([x**2 for x in (point - center)]))

    WSSSE = parsedData.map(lambda point: error(point)).reduce(lambda x, y: x + y)
    print("Within Set Sum of Squared Error = " + str(WSSSE))

    # Save and load model
    clusters.save(sc, "file:E:/Work/Python1/work/spark/anomalydetectionspark/test/spark-warehouse")

有人可以帮我找到我收到错误的原因吗？

Answer 1

您正在使用的模型的绝对路径不正确，而不是使用 file：/// E：/ Work ... 前缀。

SELECT 
    Person.Name
  , Saldo1.Duration AS Holiday
  , Saldo2.Duraten AS Hours
  , b.Amount
FROM Person_Saldo AS Saldo1 
JOIN Person_Saldo AS Saldo2 ON Saldo1.Person_ID = Saldo2.Person_ID
JOIN Person ON Saldo1.Person_ID = Person.ID
outer apply (
  SELECT Count([Date]) AS Amount 
  FROM Booking as i
  Where i.Person_Id = Saldo1.PersonId
    AND i.Timekonto = 54 
    AND i.Booking_OK = 1 
  ) as b 
WHERE Person.Group_ID= '1' 
  AND Person.Active = 'true' 
  AND Saldo1.Year = '2017' 
  AND Saldo1.Timekonto = '4' 
  AND Saldo2.Timekonto='52'

Answer 2

部署模式必须是“客户端”才能起作用，否则必须在服务器上找到传递的“本地路径”，而该服务器将没有类似于Windows路径的路径。如果部署模式为“客户端”，则必须注意，保存不会发生在并行代码块中，该代码块将在worker上执行，并且再次没有此路径。参见：https://spark.apache.org/docs/latest/submitting-applications.html

如何将spark mllib模型存储到本地文件系统（windows）

2 个答案: