我正在获取：java.lang.NoSuchMethodException：org.apache.hadoop.mapreduce.lib.output.BindingPathOutputCommitter。（） 运行spark提交时出错，其中使用用户定义的hadoop预先构建的spark版本2.4.4。 我正在将Hadoop 3.1.2与SPARK_DIST_CLASSPATH = $（path_to_hadoop-3.1.2 / bin / hadoop classpath）一起使用 在spark-env.sh中提到，外部包含的jar是aws-java-sdk-bundle-1.11.271.jar 和hadoop-aws-3.1.2.jar。
某些配置设置为使用S3Aoutput committer
spark_conf = SparkConf().set("mapreduce.fileoutputcommitter.algorithm.version", "2")
sc = SparkContext(conf=spark_conf, appName="sampleJob")
sqlContext = SQLContext(sc)
# setting the configurations to read and write from s3
sc._jsc.hadoopConfiguration().set('spark.sql.sources.commitProtocolClass',
'com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol')
sc._jsc.hadoopConfiguration().set('spark.sql.parquet.output.committer.class',
'org.apache.hadoop.mapreduce.lib.output.BindingPathOutputCommitter')
sc._jsc.hadoopConfiguration().set('spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a',
'org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory')
sc._jsc.hadoopConfiguration().set('spark.hadoop.fs.s3a.committer.name', 'partitioned')
sc._jsc.hadoopConfiguration().set('fs.s3a.committer.staging.conflict-mode', 'fail')
sc._jsc.hadoopConfiguration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
以下是追溯：
An error occurred while calling o81.parquet.
: java.lang.NoSuchMethodException: org.apache.hadoop.mapreduce.lib.output.BindingPathOutputCommitter.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol.setupCommitter(SQLHadoopMapReduceCommitProtocol.scala:62)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:161)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:139)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:566)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)