Spark 2.3.0 SQL无法将数据插入hive hbase表

时间:2018-06-04 11:51:10

标签: apache-spark hive apache-spark-sql hbase

使用与hive 2.2.0集成的Spark 2.3 thriftserver。从火花直线跑。尝试将数据插入hive hbase表(以hbase作为存储的hive表)。插入hive本地表是可以的。插入hive hbase表时,会抛出以下异常:

  

ClassCastException:org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat无法强制转换为org.apache.hadoop.hive.ql.io.HiveOutputFormat

     

at org.apache.spark.scheduler.Task.run(Task.scala:109)          在org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:345)         在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)          at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:624)          在java.lang.Thread.run(Thread.java:748)   引起:java.lang.ClassCastException:org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat无法强制转换为org.apache.hadoop.hive.ql.io.HiveOutputFormat          在org.apache.spark.sql.hive.execution.HiveFileFormat $$ anon $ 1.outputFormat $ lzycompute(HiveFileFormat.scala:93)           写行时k失败了。         在org.apache.spark.sql.execution.datasources.FileFormatWriter $ .org $ apache $ spark $ sql $ execution $ datasources $ FileFormatWriter $$ executeTask(FileFormatWriter.scala:285)         在org.apache.spark.sql.execution.datasources.FileFormatWriter $$ anonfun $ write $ 1.apply(FileFormatWriter.scala:197)         在org.apache.spark.sql.execution.datasources.FileFormatWriter $$ anonfun $ write $ 1.apply(FileFormatWriter.scala:196)        在org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)         在org.apache.spark.scheduler.Task.run(Task.scala:109)        在org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:345)        在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:624)         在java.lang.Thread.run(Thread.java:748)   引起:java.lang.ClassCastException:org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat无法强制转换为org.apache.hadoop.hive.ql.io.HiveOutputFormat        在org.apache.spark.sql.hive.execution.HiveFileFormat $$ anon $ 1.outputFormat $ lzycompute(HiveFileFormat.scala:93)

1 个答案:

答案 0 :(得分:0)

似乎是一个已知的蜂巢问题。有几个拉动请求解决了它,但我还没有找到实际修复。

https://issues.apache.org/jira/browse/SPARK-6628