Question

我正在尝试使用GridCV在Scala中进行超参数调整。但是，我创建了管道，并且所有内容都得到了满足，我将数据集适合了管道，也很适合。

然后我添加一些paramGrid，经过4个阶段后进行交叉验证，这会给我错误：

scala> val cvModel = cv.fit(df)
Tracker started, with env={DMLC_NUM_SERVER=0, DMLC_TRACKER_URI=10.xx.xx.xxx, DMLC_TRACKER_PORT=9091, DMLC_NUM_WORKER=1}
Tracker started, with env={DMLC_NUM_SERVER=0, DMLC_TRACKER_URI=10.xx.xx.xxx, DMLC_TRACKER_PORT=9091, DMLC_NUM_WORKER=1}
Tracker started, with env={DMLC_NUM_SERVER=0, DMLC_TRACKER_URI=10.xx.xx.xxx, DMLC_TRACKER_PORT=9091, DMLC_NUM_WORKER=1}
Tracker started, with env={DMLC_NUM_SERVER=0, DMLC_TRACKER_URI=10.xx.xx.xxx, DMLC_TRACKER_PORT=9091, DMLC_NUM_WORKER=1}
19/02/13 09:16:33 WARN TaskSetManager: Lost task 2.0 in stage 152.0 (TID 916, ip-10.xx.xx.xxx.ec2.internal, executor 7): java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.spark.ml.linalg.DenseVector.apply(Vectors.scala:448)
        at org.apache.spark.ml.evaluation.BinaryClassificationEvaluator$$anonfun$1.apply(BinaryClassificationEvaluator.scala:82)
        at org.apache.spark.ml.evaluation.BinaryClassificationEvaluator$$anonfun$1.apply(BinaryClassificationEvaluator.scala:81)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
        at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter..

然后出现两三个错误。我无法弄清楚为什么会发生这种情况，因为我是第一次在Scala中进行编码。但是根据我的概念和示例中给出的代码，它似乎没有用。

这是我的代码：

import java.util.Calendar
import org.apache.log4j.{Level, Logger}
import org.apache.spark.ml.feature._
import org.apache.spark.sql._
import org.apache.spark.sql.functions.lit
import java.io.PrintWriter
import java.io.File
import org.apache.spark.ml.feature.StringIndexer
import org.apache.spark.ml.tuning._
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
import ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier
import ml.dmlc.xgboost4j.scala.spark.XGBoostClassificationModel
import org.apache.spark.ml.{Pipeline, PipelineModel}

val spark = SparkSession.builder().getOrCreate()
val dataset = spark.sql("select * from userdb.xgb_train_data")
val df = dataset.na.fill(0)




val header = df.columns.filter(_ != "id").filter(_ != "y_val")
val assembler = new VectorAssembler().setInputCols(header).setOutputCol("features")

val booster= new XGBoostClassifier().setLabelCol("y_val")
val pipeline = new Pipeline().setStages(Array(assembler,booster))
val model = pipeline.fit(df)

val evaluator = new BinaryClassificationEvaluator().setLabelCol("y_val")

val paramGrid = new ParamGridBuilder().
    addGrid(booster.maxDepth, Array(3, 8)).
    addGrid(booster.eta, Array(0.2, 0.6)).
    build()


val cv = new CrossValidator().
    setEstimator(pipeline).
    setEvaluator(evaluator).
    setEstimatorParamMaps(paramGrid).
    setNumFolds(10)
val cvModel = cv.fit(df)

val bestModel = cvModel.bestModel.asInstanceOf[PipelineModel].stages()
    .asInstanceOf[XGBoostClassificationModel]
bestModel.extractParamMap()

或者还有其他方法可以进行超参数调整和交叉验证测试吗？我在执行setEvaluator代码时遇到问题。我了解的是我的特征形状和y预测形状不匹配。但是我如何确定它们呢？

P.S。我正在EMR群集上运行它。我也只是通过将算法更改为Logistic回归来尝试了同样的事情，并且工作正常。我正在使用xgboost v0.8，spark是v2.2

Answer 1

解决这个问题的诀窍是使用multiclassclassification评估程序而不是Binaryclassificationevaluator。

WARN TaskSetManager：丢失的任务xxx：java.lang.ArrayIndexOutOfBoundsException：1-Scala

1 个答案: