当我执行以下简单的Spark代码时:
val trial = Try {
require(
requirement = false,
"error!"
)
}
val ee = trial.failed.get
sparkContext.parallelize(Seq(ee))
.collect()
我遇到以下错误:
Job aborted due to stage failure: Task 3 in stage 0.0 failed 4 times, most recent failure: Lost task 3.3 in stage 0.0 (TID 6, 10.0.9.145, executor 1): com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
suppressedExceptions (java.lang.IllegalArgumentException)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:366)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:307)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:315)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:386)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at java.util.Collections$UnmodifiableCollection.size(Collections.java:1030)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:83)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
... 10 more
显然,此错误是由我在调试中发现为emptyException的成员引起的。此外,我在互联网上找不到任何类似的错误。该错误的原因是什么,我应该怎么做才能避免它?