分区表上的动态加载问题

时间:2019-06-12 06:46:13

标签: hadoop hive

动态加载到配置单元中的分区表时出错。

首先,我创建了一个未分区的表dept,如下所示:

create table dept(id int, name string) row format delimited fields terminated by ',';   

将数据加载到表中

load data inpath '/data/sample' into table sample2;

之后,我创建了一个分区表dept_part:

create table dept_part (id int) partitioned by (dept_name string) row format delimited fields terminated by ','; 

然后,当我尝试将数据从dept表加载到dept_part中时出现错误。

以下是动态加载时出现的错误:

insert overwrite table dept_part PARTITION (dept_name) select * from dept;
  

MapReduce作业总数= 3个启动作业,共3个,减少的数量   由于没有reduce运算符,所以将tasks设置为0,Jobing Job =   job_201906112238_0008,跟踪URL =   http://localhost:50030/jobdetails.jsp?jobid=job_201906112238_0008杀死   命令= /home/itelligence/hadoop-1.2.0/libexec/../bin/hadoop作业   -kill job_201906112238_0008第1阶段的Hadoop作业信息:映射器数量:1;减速机数量:0 2019-06-11 23:33:17,839第一阶段   地图= 0%,减少= 0%2019-06-11 23:33:21,869第1阶段地图= 100%,   减少= 0%,累积CPU 0.73秒2019-06-11 23:33:22,879 Stage-1   映射= 100%,减少= 0%,累积CPU 0.73秒2019-06-11   23:33:23,885第1阶段映射= 100%,减少= 0%,累积CPU 0.73秒   2019-06-11 23:33:24,893阶段1映射= 100%,减少= 100%,累积   CPU 0.73秒MapReduce总累积CPU时间:730毫秒结束的作业=   job_201906112238_0008 Stage-4由条件解析器选择。   条件解析器会过滤掉Stage-3。第5阶段已被滤除   通过条件解析器。将数据移至:   hdfs://本地主机:8020 / tmp / hive-itelligence / hive_2019-06-11_23-33-13_192_4043068268773993996-1 / -ext-10000   将数据加载到表default.dept_part分区(dept_name = null)   失败失败   MetaException(message:javax.jdo.JDODataStoreException:必需的表   缺少:目录““架构”“中的” PARTITIONS“。 DataNucleus需要   该表执行其持久性操作。您的元数据   不正确,或者您需要在以下位置启用“ datanucleus.autoCreateTables”   org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:461)     在   org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)     在   org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)     在   org.apache.hadoop.hive.metastore.ObjectStore.addPartition(ObjectStore.java:1160)     在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     在   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     在java.lang.reflect.Method.invoke(Method.java:622)在   org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)     在com.sun.proxy。$ Proxy10.addPartition(未知来源)处   org.apache.hadoop.hive.metastore.HiveMetaStore $ HMSHandler.append_partition_common(HiveMetaStore.java:1543)     在   org.apache.hadoop.hive.metastore.HiveMetaStore $ HMSHandler.append_partition_with_environment_context(HiveMetaStore.java:1598)     在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     在   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     在java.lang.reflect.Method.invoke(Method.java:622)在   org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)     在   com.sun.proxy。$ Proxy12.append_partition_with_environment_context(未知   来源)   org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(HiveMetaStoreClient.java:411)     在   org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(HiveMetaStoreClient.java:405)     在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     在   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     在java.lang.reflect.Method.invoke(Method.java:622)在   org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)     在com.sun.proxy。$ Proxy13.appendPartition(未知来源)处   org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1609)     在   org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1230)     在   org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:1412)     在org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:354)     在org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)在   org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)     在org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)在   org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)在   org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)在   org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)在   org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)     在   org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)     在   org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)     在   org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)     在org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)处   org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)在   sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)位于   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     在   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     在java.lang.reflect.Method.invoke(Method.java:622)在   org.apache.hadoop.util.RunJar.main(RunJar.java:160)   NestedThrowablesStackTrace:缺少必需的表:中的“ PARTITIONS”   目录“”架构“”。 DataNucleus要求该表执行其   持久性操作。您的元数据不正确,或者您需要   启用“ datanucleus.autoCreateTables”   org.datanucleus.store.rdbms.exceptions.MissingTableException:必需   缺少表:目录““架构”“中的” PARTITIONS“。数据核   要求该表执行其持久性操作。要么你   元数据不正确,或者您需要启用   “ datanucleus.autoCreateTables”位于   org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:485)     在   org.datanucleus.store.rdbms.RDBMSStoreManager $ ClassAdder.performTablesValidation(RDBMSStoreManager.java:3387)     在   org.datanucleus.store.rdbms.RDBMSStoreManager $ ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:3199)     在   org.datanucleus.store.rdbms.RDBMSStoreManager $ ClassAdder.run(RDBMSStoreManager.java:2850)     在   org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:122)     在   org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:1624)     在   org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:953)     在   org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:687)     在   org.datanucleus.store.rdbms.RDBMSStoreManager.getPropertiesForGenerator(RDBMSStoreManager.java:2059)     在   org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1364)     在   org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3727)     在   org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.java:2574)     在   org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOStateManager.java:526)     在   org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:202)     在   org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNew(ExecutionContextImpl.java:1326)     在   org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2123)     在   org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1972)     在   org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1820)     在   org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)     在   org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)     在   org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)     在   org.apache.hadoop.hive.metastore.ObjectStore.addPartition(ObjectStore.java:1160)     在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     在   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     在java.lang.reflect.Method.invoke(Method.java:622)在   org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)     在com.sun.proxy。$ Proxy10.addPartition(未知来源)处   org.apache.hadoop.hive.metastore.HiveMetaStore $ HMSHandler.append_partition_common(HiveMetaStore.java:1543)     在   org.apache.hadoop.hive.metastore.HiveMetaStore $ HMSHandler.append_partition_with_environment_context(HiveMetaStore.java:1598)     在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     在   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     在java.lang.reflect.Method.invoke(Method.java:622)在   org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)     在   com.sun.proxy。$ Proxy12.append_partition_with_environment_context(未知   来源)   org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(HiveMetaStoreClient.java:411)     在   org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(HiveMetaStoreClient.java:405)     在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     在   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     在java.lang.reflect.Method.invoke(Method.java:622)在   org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)     在com.sun.proxy。$ Proxy13.appendPartition(未知来源)处   org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1609)     在   org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1230)     在   org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:1412)     在org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:354)     在org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)在   org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)     在org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)在   org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)在   org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)在   org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)在   org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)     在   org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)     在   org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)     在   org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)     在org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)处   org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)在   sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)位于   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     在   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     在java.lang.reflect.Method.invoke(Method.java:622)在   org.apache.hadoop.util.RunJar.main(RunJar.java:160))失败:   执行错误,返回代码1   org.apache.hadoop.hive.ql.exec.MoveTask MapReduce启动的作业:作业   0:映射:1累积CPU:0.73秒HDFS读取:293 HDFS写入:24   成功使用MapReduce的CPU时间总计:730毫秒

有人在我做错事情的地方可以帮助我吗?

0 个答案:

没有答案