大数据-对于小数据,向phoenix的数据加载成功,但对大数据则失败

时间:2018-08-23 13:13:40

标签: performance hive hbase bigdata phoenix

我需要使用

将Hive表上传到Hive-Phoenix整数表
TBLPROPERTIES ("phoenix.table.name" = "PHOENIX_SCHEMA.SAMPLE_TBL_NAME")

因此,在成功创建表之后,我将使用以下内容来插入

INSERT INTO PHOENIX_SCHEMA.SAMPLE_TBL_NAME select * from
hive_schema.sample_tbl_name;

如果表很小(5k到2M),则说明上传成功,

如果大小过大(> 20M),上传失败,并出现以下错误。

Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
        ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
        ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
        ... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:137)
        ... 22 more
used by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.derby.jdbc.AutoloadedDriver40
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556)
        at java.sql.DriverManager.getConnection(DriverManager.java:661)
        at java.sql.DriverManager.getConnection(DriverManager.java:208)
        at org.apache.phoenix.hive.util.PhoenixConnectionUtil.getConnection(PhoenixConnectionUtil.java:96)
        at org.apache.phoenix.hive.util.PhoenixConnectionUtil.getInputConnection(PhoenixConnectionUtil.java:63)
        at org.apache.phoenix.hive.PhoenixSerializer.<init>(PhoenixSerializer.java:70)
        at org.apache.phoenix.hive.PhoenixSerDe.initialize(PhoenixSerDe.java:86)
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:369)
        at org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.initializeOp(VectorFileSinkOperator.java:74)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
        at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.initializeMapOperator(VectorMapOperator.java:580)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)
        ... 22 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

已经使用以下优化技术:

set phoenix.upsert.batch.size=10000;
set hive.fetch.task.conversion=more;
set hive.hbase.wal.enabled=false;
set hive_schema.sample_tbl_name.disable.wal=true;
set hive_schema.sample_tbl_name.auto.flush=true;
set hbase.zookeeper.quorum=xxx.xxx.xxx, xxx.xxx.xxx.xxx;

能做些什么来解决这个问题吗?

0 个答案:

没有答案