我正在将Spark（2.4.4）数据帧写入S3，并偶尔出现java.io.IOException：无法重命名S3AFileStatus错误。 Spark是在hadoop 3.1.3上捆绑安装的。
该错误是已知的，并且是由S3最终一致性引起的，这意味着可能立即出现不一致的情况， 例如在HADOOP-14161中有描述。
但是我确实使用S3guard，详细的调试日志显示它正在工作。我得到了DynamoDB表中的所有记录，DynamoDB指标显示了读写，s3guard信息显示了它已启用，等等。
我知道默认文件提交器不理想，但是我不应该避免此问题，因为S3guard可以保证即时一致性吗？
换句话说，尽管有了S3guard，为什么仍然出现错误？ 预先感谢您的宝贵时间！
WARN org.apache.spark.scheduler.TaskSetManager - Lost task 49.0 in stage 936.0 (TID 72604, 10.0.62.191, executor 0): org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to rename S3AFileStatus{path=s3a://my-spark-bucket/persisted_data/00/c9d61ba03f114d5f937eb8bdbe9ac2bc__9d6ba987-06e1-4e66-be88-f518ede26a4c_/rtg.parquet/_temporary/0/_temporary/attempt_20200123150921_0936_m_000049_72604/part-00049-f9ef3dcb-0928-466d-a663-39f023f401b8-c000.snappy.parquet; isDirectory=false; length=180872; replication=1; blocksize=33554432; modification_time=1579792215000; access_time=0; owner=root; group=root; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE to s3a://my-spark-bucket/persisted_data/00/c9d61ba03f114d5f937eb8bdbe9ac2bc__9d6ba987-06e1-4e66-be88-f518ede26a4c__Export/rtg.parquet/part-00049-f9ef3dcb-0928-466d-a663-39f023f401b8-c000.snappy.parquet
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:475)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:488)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:605)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:568)
at org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:225)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
... 10 more