Question

我有以下代码：

val launcher = new SparkLauncher()
.setSparkHome(SparkConfig.sparkHome)
.setAppResource(SparkConfig.sparkJobJar)
.setMainClass(SparkConfig.sparkJobMainClass)
.setMaster(SparkConfig.sparkMaster)
.setConf("spark.driver.extraJavaOptions", SparkConfig.sparkJobSystemProperties)
.setConf("spark.executor.extraJavaOptions", SparkConfig.sparkJobSystemProperties)
.setConf("spark.executor.memory", SparkConfig.sparkExecutorMemory)
.setConf("spark.driver.memory", SparkConfig.sparkDriverMemory)
.setConf("spark.executor.cores", SparkConfig.sparkExecutorCores)      
.setConf("spark.network.timeout", SparkConfig.sparkNetworkTimeout)          
.setConf("spark.app.name", s"my-app")
.setVerbose(true)
.addAppArgs(processId)

val handle =  launcher.startApplication(JobListenerStatus)
//waiting for the lock releases.
isFinishedLock.acquire()
handle.getState match {
 case State.FAILED | State.KILLED => {
     logger.error(String.format("Process %s: Error while processing a job spark task, job finished with %s state. Please refer to job-manager logs to more information...", processId, handle.getState().name()))
     throw new SparkJobExecutionException(handle.getState)
 }
 case _ => {
     logger.info(String.format("Process %s: Finished with sucess.", processId))
 }
}

在我的火花工作中，我有，例如：

def main(args: Array[String]): Unit = {
try {
     doTheJob Tasks...... and simulate an error
} catch {
  case t: Throwable =>
     log.error("errors")
     throw t                    
}

}

这是我的JobListenerStatus

object JobListenerStatus extends SparkAppHandle.Listener {

private def info(handle: SparkAppHandle): Unit = {

  val appId:String = Option(handle.getAppId : String).getOrElse("Not set")
  logger.info(String.format("Process %s: Yarn Application id %s with current state %s.", processId, appId, handle.getState.name()))
}

override def infoChanged(handle: SparkAppHandle): Unit = {
  info(handle)
}

override def stateChanged(handle: SparkAppHandle): Unit = {
  info(handle)
  if (handle.getState.isFinal) {
    isFinishedLock.release()
  }
}

}

问题是，即使在我的程序中抛出错误（抛出t），作业的最终状态仍为状态FINISHED。如何抛出异常以获得FAILED状态。

最终状态记录：

具有当前状态的纱线申请ID application_1475523421187_9534 结束。

我正在使用以下课程：

org.apache.spark.launcher.SparkAppHandle
org.apache.spark.launcher.SparkLauncher

事先提前。

如何在Spark中抛出异常

0 个答案: