Question

我正在尝试运行MLLIB的随机森林模型，并且遇到一些超出范围的例外：

15/09/15 01:53:56 INFO scheduler.DAGScheduler: ResultStage 5 (collect at DecisionTree.scala:977) finished in 0.147 s
15/09/15 01:53:56 INFO scheduler.DAGScheduler: Job 5 finished: collect at DecisionTree.scala:977, took 0.161129 s
15/09/15 01:53:57 INFO rdd.MapPartitionsRDD: Removing RDD 4 from persistence list
15/09/15 01:53:57 INFO storage.BlockManager: Removing RDD 4
Traceback (most recent call last):
  File "/root/random_forest/random_forest_spark.py", line 142, in <module>
    main()
  File "/root/random_forest/random_forest_spark.py", line 121, in main
    trainModel(dset)
  File "/root/random_forest/random_forest_spark.py", line 136, in trainModel
    impurity='gini', maxDepth=4, maxBins=32)
  File "/root/spark/python/lib/pyspark.zip/pyspark/mllib/tree.py", line 352, in trainClassifier
  File "/root/spark/python/lib/pyspark.zip/pyspark/mllib/tree.py", line 270, in _train
  File "/root/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 128, in callMLlibFunc
  File "/root/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 121, in callJavaFunc
  File "/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o47.trainRandomForestModel.
: java.lang.IndexOutOfBoundsException: 1337 not in [0,1337)
        at breeze.linalg.SparseVector$mcD$sp.apply$mcD$sp(SparseVector.scala:74)
        at breeze.linalg.SparseVector$mcD$sp.apply(SparseVector.scala:73)
        at breeze.linalg.SparseVector$mcD$sp.apply(SparseVector.scala:49)
        at breeze.linalg.TensorLike$class.apply$mcID$sp(Tensor.scala:94)
        at breeze.linalg.SparseVector.apply$mcID$sp(SparseVector.scala:49)
        at org.apache.spark.mllib.linalg.Vector$class.apply(Vectors.scala:102)
        at org.apache.spark.mllib.linalg.SparseVector.apply(Vectors.scala:636)
        at org.apache.spark.mllib.tree.DecisionTree$$anonfun$26.apply(DecisionTree.scala:992)
        at org.apache.spark.mllib.tree.DecisionTree$$anonfun$26.apply(DecisionTree.scala:992)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
        at org.apache.spark.mllib.tree.DecisionTree$.findSplitsBins(DecisionTree.scala:992)
        at org.apache.spark.mllib.tree.RandomForest.run(RandomForest.scala:151)
        at org.apache.spark.mllib.tree.RandomForest$.trainClassifier(RandomForest.scala:289)
        at org.apache.spark.mllib.api.python.PythonMLLibAPI.trainRandomForestModel(PythonMLLibAPI.scala:666)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
        at py4j.Gateway.invoke(Gateway.java:259)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:207)
        at java.lang.Thread.run(Thread.java:745)

我使用正确运行的data / mllib / sample_libsvm_data.txt运行示例python代码here。但是，当我使用自己的RDD时，我得到上述错误。我的RDD条目的格式是来自mllib的LabeledPoint，而每个标记点的标记由mllib SparseVector描述。我正在从numpy csr矩阵加载sparsevectors的数据。

我没有看到加载样本数据和我自己的数据有太大差别。但我确实注意到错误似乎总是在我的RDD的最后一个元素上调用。

修改在随机森林上训练我的数据的示例测试用例产生了以下错误：

py4j.protocol.Py4JJavaError: An error occurred while calling o46.trainRandomForestModel.
: java.lang.IndexOutOfBoundsException: 1071 not in [0,1071)

然后我尝试使用以下内容查看更多数据：

>>> dset = data.collect()
>>> dset[-1].features.size
1721

每个条目都是以下类型：

>>> type(dset[-1].features)
<class 'pyspark.mllib.linalg.SparseVector'>

dset[-1]的输出格式为：

LabeledPoint(0.0, (2286,[44673,64508,65588,122081,306819,306820,382530,401432,465330,465336,505179,512444,512605,517844,526648,595536,595540,615236,628547,629226,810553,938019,1044478,1232743,... ... ...],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,... ... .. ]))

请注意，功能数量的大小与错误消息的索引相同。

Answer 1

我找到了我收到这些错误的原因，所以我在这里发布它，以防其他人遇到它。

tl; dr我为SparseVector的大小存储了错误的值。

我的MLLIB LabeledPoint对象实例包含label和features，其中features应该是SparseVector对象。使用SparseVector(vector_size, nonzero_indices, data)声明此稀疏对象。

但是，我偶然使用了非零值的数量为vector_size。这可以在我的示例LabeledPoint输出LabeledPoint(0.0, (2286,[44673,64508, ...

中看到

在这里我们可以看到我将我的大小声明为2286，但是即使我的第一个索引（44673）大于我声明的数组大小，因此让我感到头痛。

将2286更改为正确的非稀疏数组大小解决了问题

培训

1 个答案: