Question

我正在尝试使用scala从一个独立的scala jar进行hive steaming。使用下面的代码段

val hiveConfig = new HiveConf
hiveConfig.addResource(new Path("file://<path to core site>"))
val hiveEP = new HiveEndPoint(hiveConfig.getVar(ConfVars.METASTOREURIS), 
<database_name>, <table_name>, new java.util.ArrayList[String])
val hConnection = hiveEP.newConnection(true,hiveConf,"")

但在以下情况遇到以下例外：

val hConnection = hiveEP.newConnection(true,hiveConf,"agent_name")

需要知道为什么会出现这种情况并解决这个问题。

我正在使用newConnection的sinature作为

public StreamingConnection newConnection(final boolean createPartIfNotExists, HiveConf conf, String agentInfo)

例外：

Exception in thread "main" com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Lorg/apache/hadoop/hive/conf/HiveConf;[Ljava/lang/Class;[Ljava/lang/Object;Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261)
        at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)
        at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227)
        at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202)
        at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:544)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:312)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:278)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:215)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:192)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:162)
        at com.example.HiveStreaming$.main(HiveStreaming.scala:33)
        at com.example.HiveStreaming.main(HiveStreaming.scala)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Lorg/apache/hadoop/hive/conf/HiveConf;[Ljava/lang/Class;[Ljava/lang/Object;Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;
        at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230)
        at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)

Answer 1

为时已晚，但可能会帮助某人。发生这种情况的原因是，如果不是全部，则依赖链中的所有程序包都在spark群集中可用。确保所有软件包均为maven jar或在spark提交配置中正确引用。

Hive Streaming异常：java.lang.NoSuchMethodError：org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy

例外：

1 个答案: