在镶木地板文件上创建外部表时,BIGSQL中的列顺序是否重要?

时间:2019-04-25 08:02:56

标签: hadoop hive bigsql

在我的Spark代码中,我将数据帧写为hdfs上的实木复合地板文件。然后,我通过更改BIGSQL中的镶木地板文件的列顺序来创建外部表,查询该表后,它向我显示以下错误。

但是,如果我在蜂巢中查询同一张表,它将起作用。我们得到逐列映射的输出。

big Parquet文件上创建外部表时,bigsql是否支持列到列映射?

BIGSQL error log: 

DB2 LOG: The statement failed because a Big SQL component encountered an error. Component receiving the error: "BigSQL IO". Component returning the error: "UNKNOWN". Log entry identifier: "[BSL-3-25371e740]".. SQLCODE=-5105, SQLSTATE=58040, DRIVER=4.22.36
Learn more about this error

BIG SQL LOG:    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsVectorizedPrimitiveColumnReader.decodeDictionaryIds(DfsVectorizedPrimitiveColumnReader.java:506)
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsVectorizedPrimitiveColumnReader.readBatch(DfsVectorizedPrimitiveColumnReader.java:173)
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsVectorizedParquetRecordReader.nextBatch(DfsVectorizedParquetRecordReader.java:428)
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsVectorizedParquetRecordReader.next(DfsVectorizedParquetRecordReader.java:347)
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsParquetSplit2Batch.split2Batch(DfsParquetSplit2Batch.java:98)
    at com.ibm.biginsights.bigsql.dfsrw.jaro.DfsSplitManager$SplitRunnable.run(DfsSplitManager.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(Thread.java:785)
    at com.ibm.biginsights.bigsql.dfsrw.jaro.DfsSplit2BatchThread.run(DfsSplit2BatchThread.java:58)
2019-04-25 03:42:14,848 ERROR com.ibm.biginsights.bigsql.dfsrw.reader.DfsBaseReader [Master-3-S:5.1001.1.0.0.724] : [BSL-3-25371e740] Exception raised by Reader at node: 3 Scan ID: S:5.1001.1.0.0.724 Table: mc400.hutdnp_ext Spark: false VORC: false VPQ: true VAVRO: false VTEXT: false VRCFILE: false VANALYZE: false
Exception Label: UNMAPPED(java.lang.NullPointerException)
java.lang.NullPointerException
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsVectorizedPrimitiveColumnReader.decodeDictionaryIds(DfsVectorizedPrimitiveColumnReader.java:506)
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsVectorizedPrimitiveColumnReader.readBatch(DfsVectorizedPrimitiveColumnReader.java:173)
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsVectorizedParquetRecordReader.nextBatch(DfsVectorizedParquetRecordReader.java:428)
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsVectorizedParquetRecordReader.next(DfsVectorizedParquetRecordReader.java:347)
    at com.ibm.biginsights.bigsql.dfsrw.reader.parquet.DfsParquetSplit2Batch.split2Batch(DfsParquetSplit2Batch.java:98)
    at com.ibm.biginsights.bigsql.dfsrw.jaro.DfsSplitManager$SplitRunnable.run(DfsSplitManager.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)


SQL statement
select * from schema_name.dummytable
Execution log
Run time: 1.275 s
Status: FAILED
Database: HDP-DEV-BIGSQL

0 个答案:

没有答案