Spark Yarn mode is not working throws java Null Pointer exception on executors stages

时间:2018-07-25 04:42:07

标签: apache-spark yarn

My Scala program does parsing the log using Java object method called parse and it works fine in local[*] mode however it is not working neither on Cloudera yarn client mode or cluster mode.

val hivehbaserows = spark.sql(" select log msg from hivehbasetable")
hivehbaserows.foreach(x => { 
                             val line = x.getString(0).toString 
                             javaObject.javaparsemethod(line)
                           }

for me when I run the above program on local[*], it works however if I launch it on yarn client or cluster mode, it throws javanullpointer exception on executors stages and throws the bellow error.

executor 6), java.lang.NullPointerException(null) [duplicate 1]
executor 2), java.lang.NullPointerException(null) [duplicate 2]
executor 2), java.lang.NullPointerException(null) [duplicate 1]
job aborted due to stage:Task 0 in stage 1.0 failed 4 times, most recent failure:lost task 0.3 in stage in 1.0 java Null Pointer exception

0 个答案:

没有答案