使用JDBC将Dataframe写入现有Hive表时出错

时间:2016-07-30 10:27:37

标签: apache-spark dataframe hive

使用jdbc将数据帧写入hive表时获取以下错误。 Hiveserver2正在运行,我可以使用beeline连接到它。 Hive日志在下面显示相同的错误消息,我没有看到有关日志的任何其他信息。

***表是使用****预先创建的

CREATE TABLE btiflag_htbl (
`clfentid` CHAR(15), 
`clfcust` CHAR(15), 
`clfflag` CHAR(2), 
`clfcrtdat` CHAR(10),
`clfdtaid` INT, 
`indgencod` CHAR(6),
`age` INT, 
`indmarsts` CHAR(6), 
`indnatcod` CHAR(6),
`quarter` CHAR(8), 
`year` INT, 
`month` INT) 
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
STORED AS PARQUET 

*将数据框写入Hive *

val prop = new Properties()
    prop.put("user", "user1")
    prop.put("password", "pwd12")
    prop.put("driver", "org.apache.hive.jdbc.HiveDriver")

    val dfWriter = btiflagDF.write.mode(SaveMode.Overwrite)

    dfWriter.jdbc("jdbc:hive2://192.168.0.10:10000/database_hive", "btiflag_htbl", prop)

****错误****

Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:36 cannot recognize input near 'INTEGER' ',' 'CLFCUST' in column type
    at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:264)
    at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:250)
    at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:309)
    at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
    at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:448)
    at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:302)
    at com.efx.btiflag_analytics$.main(btiflag_analytics.scala:81)
    at com.efx.btiflag_analytics.main(btiflag_analytics.scala)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:36 cannot recognize input near 'INTEGER' ',' 'CLFCUST' in column type
    at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
    at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:112)
    at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
    at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
    at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388)
    at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
    at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
    at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
    at com.sun.proxy.$Proxy20.executeStatementAsync(Unknown Source)
    at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274)
    at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
    at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
    at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:36 cannot recognize input near 'INTEGER' ',' 'CLFCUST' in column type
    at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:205)
    at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:396)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
    at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
    at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1116)
    at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
    ... 27 more

任何帮助将不胜感激

0 个答案:

没有答案