当Spark运行特定阶段时,杀死Spark执行器

时间:2018-07-05 18:37:58

标签: scala apache-spark hadoop

我在YARN上运行火花。我的代码很简单。我想在执行“ data.repartition(10)”时杀死一名执行者。我可以轻松地做到吗?

val data = sc.sequenceFile[NullWritable, BytesWritable](inputPath)
.map { case (key, value) =>
    Data.fromBytes(value)
}

var process = data.repartition(10) // kill one executor here
process.map { d =>
    val data = d.toByteArray
    (new AvroKey(ByteBuffer.wrap(data)), NullWritable.get())
}
.saveAsNewAPIHadoopFile[AvroKeyOutputFormat[ByteBuffer]](outputPath)

0 个答案:

没有答案