Question

向aws s3存储桶写入时，火花作业失败，并且我正在获取java.io.FileNotFoundException：没有这样的文件或目录

堆栈跟踪：

java.io.FileNotFoundException: No such file or directory: s3a://vishal/test/abc.parquet/_temporary/0/task_20190422091705_0001_m_000000 
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:993) 
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:734) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310) 
at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48) 
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:166) 
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:185) 
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155) 
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) 
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) 
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) 
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80) 
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80) 
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668) 
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668) 
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78) 
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125) 
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73) 
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668) 
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276) 
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270) 
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:228) 
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:557)

Answer 1

很可能是权限问题。从您的异常来看，Spark作业似乎缺少对GetObject的AWS S3权限。确保您的Spark Job至少具有读取，写入和移动权限。

但是，如果您正在使用Spark写入S3位置，并希望该文件可立即用于Spark，则由于每次都无法正常工作，因为

S3可能需要一些时间才能使新创建的对象出现在目录列表中，而删除的对象可能仍然可见。

如果您仍然需要这样做，我建议您在文件最终移动到S3之后，在可靠的持久性存储（例如本地（例如磁盘或HDFS））上执行所有与文件相关的操作。

但是，如果您没有其他选择，请尝试S3Guard。

写入AWS S3存储桶时Spark作业失败-

1 个答案: