我已经开始了Metastore和hiveserver2
#./hive --service metastore
#./hive --service hiveserver2
当我在查询下面执行时
#./beeline -u jdbc:hive2://192.168.0.10:10000 -e 'select count(*) from test_tb' --hiveconf hive.root.logger=DEBUG,console --verbose=true
抛出错误
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:275)
at org.apache.hive.beeline.Commands.execute(Commands.java:736)
at org.apache.hive.beeline.Commands.sql(Commands.java:657)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:804)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:608)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:630)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Beeline version 0.13.1 by Apache Hive
hiveserver2登录
6/06/14 10:57:32 [main]: WARN common.LogUtils: DEPRECATED: Ignoring hive-default.xml found on the CLASSPATH at /data/offline/apache-hive-0.13.1-bin/conf/hive-default.xml
16/06/14 10:57:32 [main]: INFO metastore.HiveMetaStore: Starting hive metastore on port 9083
16/06/14 10:57:32 [main]: INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/06/14 10:57:32 [main]: INFO metastore.ObjectStore: ObjectStore, initialize called
16/06/14 10:57:33 [main]: INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/06/14 10:57:33 [main]: INFO metastore.ObjectStore: Initialized ObjectStore
16/06/14 10:57:34 [main]: INFO metastore.HiveMetaStore: Added admin role in metastore
16/06/14 10:57:34 [main]: INFO metastore.HiveMetaStore: Added public role in metastore
16/06/14 10:57:34 [main]: INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
16/06/14 10:57:34 [main]: INFO metastore.HiveMetaStore: Starting DB backed MetaStore Server
16/06/14 10:57:34 [main]: INFO metastore.HiveMetaStore: Started the new metaserver on port [9083]...
16/06/14 10:57:34 [main]: INFO metastore.HiveMetaStore: Options.minWorkerThreads = 200
16/06/14 10:57:34 [main]: INFO metastore.HiveMetaStore: Options.maxWorkerThreads = 100000
16/06/14 10:57:34 [main]: INFO metastore.HiveMetaStore: TCP keepalive = true
16/06/14 10:57:40 [pool-3-thread-1]: INFO metastore.HiveMetaStore: 1: source:/10.234.177.127 get_table : db=default tbl=test_tb
16/06/14 10:57:40 [pool-3-thread-1]: INFO HiveMetaStore.audit: ugi=qspace ip=/10.234.177.127 cmd=source:/192.168.0.10 get_table : db=default tbl=test_tb
16/06/14 10:57:40 [pool-3-thread-1]: INFO metastore.HiveMetaStore: 1: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/06/14 10:57:40 [pool-3-thread-1]: INFO metastore.ObjectStore: ObjectStore, initialize called
16/06/14 10:57:40 [pool-3-thread-1]: INFO metastore.ObjectStore: Initialized ObjectStore
答案 0 :(得分:2)
我看到你正在使用Hiveserver2。对于这样的聚合,根据配置,我认为您可能需要在执行之前设置reducer的数量。使用Hive,您可以使用以下语法:
SET mapred.reduce.tasks=1
但是在Hive2上我注意到我需要使用:
SET mapreduce.job.reduces=1
我希望这有帮助!以前我有一个相同的错误消息,并改变这解决了我的问题。
答案 1 :(得分:1)
这可能是权限问题,请尝试按以下方式启动beeline
直线
!连接jdbc:hive2:// hadoop7:10000 / default
输入您的用户名和密码
答案 2 :(得分:0)
在运行查询之前添加以下代码段:
SET hive.auto.convert.join=false;
SET mapreduce.map.memory.mb = 16384;
SET mapreduce.map.java.opts='-Djava.net.preferIPv4Stack=true -Xmx13107M';
SET mapreduce.reduce.memory.mb = 13107;
SET mapreduce.reduce.java.opts='-Djava.net.preferIPv4Stack=true -Xmx16384M';
set hive.support.concurrency = false;
答案 3 :(得分:0)
确认在我的情况下,这是由权限问题引起的(如wise_w所指出的),很奇怪的是,它看起来像普通select *可以正常工作,但是任何更复杂的方法都没有。
我尝试了以下操作:
$ beeline -u jdbc:hive2://server1:10000
(connected fine, no errors)
> select * from table1;
-> worked fine
> select count(*) from table1;
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)
or
ERROR : Job Submission failed with exception 'org.apache.hadoop.security.AccessControlException(Permission denied: user=anonymous, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
解决方案始于wise_w建议
$ beeline --incremental=true
$ !connect jdbc:hive2://server1:10000/default
Enter username for jdbc:hive2://server1:10000/default: hive
Enter password for jdbc:hive2://server1:10000/default: **** (this is hive as well in my test setup)
->现在一切正常。
现在有一支班轮
$ beeline -u jdbc:hive2://server1:10000 -n hive -p hive --incremental=true
我觉得冲突来自事实,就像这样的联系
$ beeline -u jdbc:hive2://server1:10000
不会引发任何错误。
答案 4 :(得分:0)
设置以下参数,在这种情况下很重要:
set mapred.job.queue.name=<your queue name>;
set mapreduce.job.reduces=1;