来自风暴蜂巢的HiveTopology问题

时间：2015-10-09 23:04:46

标签： hive apache-storm

我正在尝试从我的本地计算机上的storm-hive项目运行HiveTopology，但是连接到在Linux服务器上运行的远程配置单元。我已按照创建ORC格式表的说明进行操作，并按照here

完成所有设置

我收到以下错误

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) ~[guava-18.0.jar:na]
    at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:229) ~[storm-hive-0.10.0.jar:0.10.0]
    at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:226) ~[storm-hive-0.10.0.jar:0.10.0]
    at org.apache.storm.hive.common.HiveWriter$9.call(HiveWriter.java:332) ~[storm-hive-0.10.0.jar:0.10.0]
    at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0_45]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_45]
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
    ... 17 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
    at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_45]
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) ~[hive-metastore-1.2.1.jar:1.2.1]
    ... 27 common frames omitted
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
    at org.apache.hadoop.security.Groups.getGroups(Groups.java:182) ~[hadoop-common-2.7.1.jar:na]
    at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1518) ~[hadoop-common-2.7.1.jar:na]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:436) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:330) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    ... 32 common frames omitted

请指出正确的方向来解决问题。 我在用 Hadoop：2.7.1 Hive：1.2.1具有事务属性。

1 个答案:

答案 0 :(得分：1)

我遇到了同样的问题，他们正在运行风暴蜂巢螺栓。根本原因如下：

Caused by: java.lang.NullPointerException: null
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) ~[na:1.8.0_20]
at org.apache.hadoop.util.Shell.runCommand(Shell.java:483) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.util.Shell.run(Shell.java:456) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:84) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:239) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:220) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:208) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) ~[guava-14.0.1.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379) ~[guava-14.0.1.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342) ~[guava-14.0.1.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257) ~[guava-14.0.1.jar:na]

org.apache.hadoop.util.Shell类中的问题。 方法getGroupsForUserCommand在Windows上运行时返回错误的命令，但找不到winutils.exe。

public static String[] getGroupsForUserCommand(final String user) {
//'groups username' command return is non-consistent across different unixes
return (WINDOWS)? new String[] { WINUTILS, "groups", "-F", "\"" + user + "\""}
                : new String [] {"bash", "-c", "id -gn " + user
                                 + "&& id -Gn " + user};}

此代码将返回包含4个元素的字符串数组，第一个元素将为null。

要解决此问题，请设置HADOOP_HOME env变量，并在dir％HADOOP_HOME％/ bin中必须为winutils.exe（下载here）。

相关问题
最新问题