Pyspark mllib +计数或收集方法引发ArrayIndexOutOfBounds异常

时间:2019-05-07 07:03:01

标签: python-3.x apache-spark pyspark apache-spark-mllib

我正在学习pyspark和mllib。

在使用RF模型预测测试数据之后,我将结果分配给称为“预测”的变量,即RDD。

如果我调用predictions.count()或prediction.collect(),则它将失败,并带有以下异常。

请您分享您的想法?已经花费了相当长的时间,但是没有找到缺少的东西。

    predictions = predict(training_data, test_data)

  File "/mp5/part_d_poc.py", line 36, in predict

    print(predictions.count())

  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1055, in count

  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1046, in sum

  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 917, in fold

  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 816, in collect

  File "/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__

  File "/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.

: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 1 times, most recent failure: Lost task 0.0 in stage 15.0 (TID 28, localhost, executor driver): java.lang.ArrayIndexOutOfBoundsException: 7

我通过以下方式构造了训练数据。

raw_training_data.map(lambda row: LabeledPoint(row.split(',')[-1], Vectors.dense(row.split(',')[0:-1])))

0 个答案:

没有答案