Question

我在1.6.3 spark版本中使用Spark HiveContext运行hive merge命令，但它失败并出现以下错误。

2017-09-11 18:30:33 Driver [INFO ] ParseDriver - Parse Completed
2017-09-11 18:30:34 Driver [INFO ] ParseDriver - Parsing command: MERGE INTO emp_with_orc AS T USING SOURCE_TABLE AS S 
ON T.id = S.id 
WHEN MATCHED AND (S.operation = 1) THEN UPDATE SET a = S.a,b = S.b 
WHEN MATCHED AND (S.operation = 2) THEN DELETE 
WHEN NOT MATCHED THEN INSERT VALUES (S.id, S.a, S.b)
2017-09-11 18:30:34 Driver [ERROR] HiveWriter - Error while executing the merge query.
org.apache.spark.sql.AnalysisException: cannot recognize input near 'MERGE' 'INTO' 'emp_with_orc'; line 1 pos 0
    at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:318)
    at org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
    at org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
    at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
    at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
    at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
    at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
    at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
    at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
    at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
    at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
    at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
    at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
    at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)

我不确定在Spark的HiveContext中是否支持ACID事务合并命令。

对此有任何帮助将不胜感激。

Answer 1

要使用MERGE操作，您需要通过HIVE JDBC执行它，因为截至目前，Spark SQL不支持MERGE。

Answer 2

Spark不支持UPDATES或DELETES，因此异常是预期的行为。

Hive Merge命令在Spark HiveContext中不起作用

2 个答案: