Question

由于以下错误，我遇到通过spark-submit运行spark作业的问题：

16/11/16 11:41:12 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map, boolean, int, boolean, boolean, boolean)
java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map, boolean, int, boolean, boolean, boolean)
at java.lang.Class.getMethod(Class.java:1786)
at org.apache.spark.sql.hive.client.Shim.findMethod(HiveShim.scala:114)
at org.apache.spark.sql.hive.client.Shim_v0_14.loadDynamicPartitionsMethod$lzycompute(HiveShim.scala:404)
at org.apache.spark.sql.hive.client.Shim_v0_14.loadDynamicPartitionsMethod(HiveShim.scala:403)
at org.apache.spark.sql.hive.client.Shim_v0_14.loadDynamicPartitions(HiveShim.scala:455)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$loadDynamicPartitions$1.apply$mcV$sp(ClientWrapper.scala:562)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$loadDynamicPartitions$1.apply(ClientWrapper.scala:562)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$loadDynamicPartitions$1.apply(ClientWrapper.scala:562)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$withHiveState$1.apply(ClientWrapper.scala:281)
at org.apache.spark.sql.hive.client.ClientWrapper.liftedTree1$1(ClientWrapper.scala:228)
at org.apache.spark.sql.hive.client.ClientWrapper.retryLocked(ClientWrapper.scala:227)
at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:270)
...

我使用带有scala 2.10的spark 1.6.0，hive 1.1.0，平台是带有spark和hive相同版本的CDH 5.7.1。在类路径上传递给spark作业的hive-exec是hive-exec-1.1.0-cdh5.7.1.jar。这个jar有一个类org.apache.hadoop.hive.ql.metadata.Hive，我可以看到它有以下方法：

public java.util.Map<java.util.Map<java.lang.String, java.lang.String>, org.apache.hadoop.hive.ql.metadata.Partition> loadDynamicPartitions(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map<java.lang.String, java.lang.String>, boolean, int, boolean, boolean, boolean) throws org.apache.hadoop.hive.ql.metadata.HiveException;

与我正在使用的库spark-hive_2.10-1.6.0.jar附带的org.apache.spark.sql.hive.client.ClientWrapper类上的那个不同，此类中相同方法的签名正在使用使用此方法的班级org.apache.spark.sql.hive.client.HiveShim：

private lazy val loadDynamicPartitionsMethod =
findMethod(
  classOf[Hive],
  "loadDynamicPartitions",
  classOf[Path],
  classOf[String],
  classOf[JMap[String, String]],
  JBoolean.TYPE,
  JInteger.TYPE,
  JBoolean.TYPE,
  JBoolean.TYPE)

我还检查了hive-exec jar的历史记录，似乎在版本1.0.0之后更改了类org.apache.hadoop.hive.ql.metadata.Hive的签名。我是Spark的新手，但在我看来，spark-hive库使用了一个旧的Hive实现（我可以在jar中的META-INF / DEPENDENCIES文件中看到已经声明了对org.spark-project.hive的依赖：蜂房EXEC：罐子：1.2.1.spark）。有谁知道如何设置spark作业以使用正确的hive库？

Answer 1

确保您已设置以下设置

SET hive.exec.dynamic.partition=true; 
SET hive.exec.max.dynamic.partitions=2048
SET hive.exec.dynamic.partition.mode=nonstrict;

在Spark中，您可以设置hive Context，如下所示

hiveCtx.setConf("hive.exec.dynamic.partition","true")
hiveCtx.setConf("hive.exec.max.dynamic.partitions","2048")
hiveCtx.setConf("hive.exec.dynamic.partition.mode", "nonstrict")

如果问题仍然存在，我猜这意味着你正在使用的火花版本与你试图运行你的spark-submit的环境不匹配...你可以尝试在spark-shell中运行你的程序，如果它工作，然后尝试将火花版本与环境设置对齐。

您可以设置对您的依赖关系，如下所示：pom

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.6.3"
libraryDependencies += "org.apache.hive" % "hive-exec" % "1.1.0"

请参考 https://mvnrepository.com/artifact/org.apache.spark

您可以使用以下命令进行环境设置 SPARK_PRINT_LAUNCH_COMMAND = true spark-shell

替代方法是使用spark partition by来保存数据

    dataframe.write.mode("overwrite").partitionBy("col1", "col2").json("//path")

由于java.lang.NoSuchMethodException：org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions，Spark作业失败

1 个答案: