Question

我有带有spark（1.6.1），hdfs和hive（2.1）的纱线簇。我的工作流程在今天工作了好几个月（代码/环境没有任何变化）。我开始遇到这样的错误：

org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 21
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
invertedWorkGraph (org.apache.hadoop.hive.ql.plan.SparkWork)
    at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:238)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:226)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:745)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:131)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
    at org.apache.hadoop.hive.ql.exec.spark.KryoSerializer.deserialize(KryoSerializer.java:49)
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:318)
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366)
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

使用配置单元我可以进行简单的选择，但是每个需要spark的操作都会在控制台中以Error: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=3)结束，并且在纱线日志中会出现错误。现在我的每个蜂巢数据库都瘫痪了（我很少）。我试图整天解决这个问题，但无法做任何事情（蜂巢重启，纱线节点的重启，更换纱线主人）。

您认为导致问题的原因是什么？如何解决？

Answer 1

我明白了。

重启hive-server2一小段时间而不是出错：TestDate MatCount1 MatCount2 Jan 2017 10 5 Feb 2017 20 0 Mar 2017 30 15 Apr 2017 0 25我收到错误：org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 26。很明显，第二种形式是节点上执行的火花在类路径上没有一些罐子。我不知道原因，为什么火花片刻无法加载这些罐子，但是在将它们手动复制到每个节点上的lib文件夹并重新启动节点后，一切都恢复正常。

org.apache.hive.com.esotericsoftware.kryo.KryoException：遇到未注册的类ID：21

1 个答案: