Hive在存储流数据时出现问题

时间:2019-01-11 17:55:43

标签: apache-spark hive apache-kafka

我有一个Kafka Streaming Consumer作业,它将数据存储在Hive表中。 问题是有时由于Hive导致作业失败,因此必须重新启动。 所有尝试访问Hive的流消费者工作都在发生这种情况。 无法追踪原因。 感谢您的建议。

下面是高级错误日志

INFO ApplicationMaster: Starting the user application in a separate Thread
INFO ApplicationMaster: Waiting for spark context initialization...
WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
WARN Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: java.lang.reflect.InvocationTargetException
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: Peer indicated failure: DIGEST-MD5: IO error acquiring password

 INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
 WARN metastore: Failed to connect to the MetaStore Server...
 INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
 WARN metastore: Failed to connect to the MetaStore Server...
 INFO metastore: Waiting 1 seconds before next connection attempt.
 INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
 WARN metastore: Failed to connect to the MetaStore Server...
 INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
 WARN metastore: Failed to connect to the MetaStore Server...
 INFO metastore: Waiting 1 seconds before next connection attempt.
 INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
 WARN metastore: Failed to connect to the MetaStore Server...
 INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
 WARN metastore: Failed to connect to the MetaStore Server...
 INFO metastore: Waiting 1 seconds before next connection attempt.
 ERROR ApplicationMaster: User class threw exception: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
Caused by: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: java.lang.reflect.InvocationTargetException
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: Peer indicated failure: DIGEST-MD5: IO error acquiring password

0 个答案:

没有答案