Flink s3读取错误:读取的数据长度与预期的长度不同

时间:2019-06-04 17:04:34

标签: amazon-s3 apache-flink

使用flink 1.7.0,但也可以在flink 1.8.0上看到。通过flink .readFile源从S3读取压缩的对象时,我们会遇到频繁但有些随机的错误:

org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException: Data read has a different length than the expected: dataLength=9713156; expectedLength=9770429; includeSkipped=true; in.getClass()=class org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client$2; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0
    at org.apache.flink.fs.s3base.shaded.com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:151)
    at org.apache.flink.fs.s3base.shaded.com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:93)
    at org.apache.flink.fs.s3base.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:76)
    at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AInputStream.closeStream(S3AInputStream.java:529)
    at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AInputStream.close(S3AInputStream.java:490)
    at java.io.FilterInputStream.close(FilterInputStream.java:181)
    at org.apache.flink.fs.s3.common.hadoop.HadoopDataInputStream.close(HadoopDataInputStream.java:89)
    at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227)
    at java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
    at org.apache.flink.api.common.io.InputStreamFSInputWrapper.close(InputStreamFSInputWrapper.java:46)
    at org.apache.flink.api.common.io.FileInputFormat.close(FileInputFormat.java:861)
    at org.apache.flink.api.common.io.DelimitedInputFormat.close(DelimitedInputFormat.java:536)
    at org.apache.flink.streaming.api.functions.source.ContinuousFileReaderOperator$SplitReader.run(ContinuousFileReaderOperator.java:336)
是 在给定的作业中,我们通常会看到许多/大多数作业都已成功读取,但是几乎总是有至少一个失败(例如,在50个文件中)。

似乎此错误实际上是由于AWS客户端引起的,所以也许flink与它无关,但是我希望有人可能对如何可靠地实现此工作有所了解。

发生错误时,最终会杀死源并取消所有连接的运算符。我还是flink的新手,但是我认为这可以从以前的快照中恢复吗?我应该期望在发生此类异常时flink重试读取文件吗?

1 个答案:

答案 0 :(得分:1)

也许您可以尝试为s3a添加更多连接,例如

flink:
...
    config: |
      fs.s3a.connection.maximum: 320