Question

I'm trying to upload a file to amazon s3. Instead of uploading, I want to read the data from database using spring batch and write the file directly into the s3 storage. Is there anyway we can do that ?

Answer 1

我有同样的事情要做。因为spring没法单独写一条流，所以我像上面的例子那样做了一个自己的事情：

您需要为此分类。一个实现WriteableResource并扩展AbstractResource的Resource类：

...

public class S3Resource extends AbstractResource implements WritableResource {

   ByteArrayOutputStream resource = new ByteArrayOutputStream();

    @Override
    public String getDescription() {
        return null;
    }

    @Override
    public InputStream getInputStream() throws IOException {
        return new ByteArrayInputStream(resource.toByteArray());
    }

    @Override
    public OutputStream getOutputStream() throws IOException {
        return resource;
    }
}

还有扩展ItemWriter的作家：

public class AmazonStreamWriter<T> implements ItemWriter<T>{

    private WritableResource resource;
    private LineAggregator<T> lineAggregator;
    private String lineSeparator;

    public String getLineSeparator() {
        return lineSeparator;
    }

    public void setLineSeparator(String lineSeparator) {
        this.lineSeparator = lineSeparator;
    }

    AmazonStreamWriter(WritableResource resource){
        this.resource = resource;
    }

    public WritableResource getResource() {
        return resource;
    }

    public void setResource(WritableResource resource) {
        this.resource = resource;
    }

    public LineAggregator<T> getLineAggregator() {
        return lineAggregator;
    }

    public void setLineAggregator(LineAggregator<T> lineAggregator) {
        this.lineAggregator = lineAggregator;
    }

    @Override
    public void write(List<? extends T> items) throws Exception {
        try (OutputStream outputStream = resource.getOutputStream()) {
                StringBuilder lines = new StringBuilder();
                Iterator var3 = items.iterator();

                while(var3.hasNext()) {
                    T item = (T) var3.next();
lines.append(this.lineAggregator.aggregate(item)).append(this.lineSeparator);
                }
                outputStream.write(lines.toString().getBytes());
        }
    }
}

使用此设置，您将编写从数据库中接收到的项目信息，并通过OutputStream将其写入Customresource。然后，可以在您的步骤之一中使用已填充的资源zu打开InputStream并通过客户端上载到S3。我这样做的原因是：amazonS3.putObject(awsBucketName, awsBucketKey , resource.getInputStream(), new ObjectMetadata());

我的解决方案可能不是完美的方法，但是从现在开始您可以对其进行优化。

Answer 2

Spring Cloud AWS adds支持Amazon S3服务，以使用资源加载器和s3协议加载和写入资源。拥有configured the AWS resource loader之后，您可以编写自定义的Spring Batch编写器，例如：

import java.io.OutputStream;
import java.util.List;

import org.springframework.batch.item.ItemWriter;
import org.springframework.core.io.ResourceLoader;
import org.springframework.core.io.WritableResource;

public class AwsS3ItemWriter implements ItemWriter<String> {

    private ResourceLoader resourceLoader;

    private WritableResource resource;

    public AwsS3ItemWriter(ResourceLoader resourceLoader, String resource) {
        this.resourceLoader = resourceLoader;
        this.resource = (WritableResource) this.resourceLoader.getResource(resource);
    }

    @Override
    public void write(List<? extends String> items) throws Exception {
        try (OutputStream outputStream = resource.getOutputStream()) {
            for (String item : items) {
                outputStream.write(item.getBytes());
            }
        }
    }
}

然后，您应该可以将此编写器与S3资源（例如s3://myBucket/myFile.log）一起使用。

反正我们可以做到吗？

请注意，我没有编译/测试先前的代码。我只是想给你一个做事的起点。

希望这会有所帮助。

Answer 3

问题是 OutputStream 只会写入该步骤发送的最后一个 列表项 。。我认为您可能需要在文件系统上写入一个临时文件，然后将整个文件发送到单独的Tasklet中

请参见以下示例： https://github.com/TerrenceMiao/AWS/blob/master/dynamodb-java/src/main/java/org/paradise/microservice/userpreference/service/writer/CSVFileWriter.java

spring batch file writer to write directly to amazon s3 storage without PutObjectRequest

3 个答案: