Question

在定义为DECIMAL（31,8）的列中加载带有NULL的ORC外部表时，我遇到了一个围绕HIVE的问题。看起来hive在加载后无法读取ORC文件，并且无法再查看该字段内的NULL记录。可以正确读取同一ORC文件中的其他记录。

这种情况最近才发生，我们的HIVE版本没有变化。令人惊讶的是，先前加载到DECIMAL字段中具有NULL的同一个表中的ORC文件是可查询的，没有问题。

我们正在使用HIVE 1.2.1。 HIVE的完整堆栈跟踪吐出来，我用

取代了实际的hdfs位置

org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.io.IOException: Error reading file: <hdfs location>
        at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:352)
        at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:220)
        at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:685)
        at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:454)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:672)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.io.IOException: Error reading file: <hdfs location>
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:507)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1670)
        at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347)
        ... 13 more
Caused by: java.io.IOException: Error reading file: <hdfs location>
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1051)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.next(OrcRawRecordMerger.java:263)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.next(OrcRawRecordMerger.java:547)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$1.next(OrcInputFormat.java:1235)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$1.next(OrcInputFormat.java:1219)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$NullKeyRecordReader.next(OrcInputFormat.java:1151)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$NullKeyRecordReader.next(OrcInputFormat.java:1137)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:474)
        ... 17 more
Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed stream Stream for column 6 kind DATA position: 201 length: 201 range: 0 offset: 289 limit: 289 range 0 = 0 to 201 uncompressed: 362 to 362
        at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
        at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
        at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1044)
        ... 24 more

Answer 1

在您的代码Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 65445 obs. of 10 variables: $ region : chr "Norte" "Norte" "Norte" "Norte" ... $ state : chr "RO" "RO" "RO" "RO" ... $ county : chr "Ariquemes" "Ariquemes" "Ariquemes" "Ariquemes" ... $ type_area : chr "Urbana" "Urbana" "Urbana" "Urbana" ... $ school_type: chr "Estadual" "Privada" "Privada" "Privada" ... $ school : chr "EEEFM PROFESSORA CARMEM IONE DE ARAUJO" "CEET - CENTRO DE EXCELENCIA EM EDUCACAO E TECNOLOGIA SENAI - JOSE FERNANDES DE MOURA" "SENAC ARIQUEMES" "COLEGIO EXCELENCIA" ... $ school_code: num 11048620 11049375 11049758 11049910 11052406 ... $ cep : num 76874026 76872838 76876674 76873524 76873743 ... $ lat : logi NA NA NA NA NA NA ... $ long : logi NA NA NA NA NA NA ...

中进行设置

HIVE - ORC读取NULL十进制值的问题 - java.io.EOFException：读取BigInteger过EOF

1 个答案: