我将AsyncHttpClient library用于异步非阻塞请求。 我的情况是：将数据通过网络接收后写入文件。
要从远程主机下载文件并保存到文件，我使用默认的ResponseBodyPartFactory.EAGER和AsynchronousFileChannel，以便在数据到达时不阻止netty线程。但是，正如我的测量结果所示，与LAZY相比，Java堆中的内存消耗增加了很多倍。
所以我决定直接进入LAZY，但没有考虑文件的后果。
此代码将有助于重现该问题。
public static class AsyncChannelWriter {
private final CompletableFuture<Integer> startPosition;
private final AsynchronousFileChannel channel;
public AsyncChannelWriter(AsynchronousFileChannel channel) throws IOException {
this.channel = channel;
this.startPosition = CompletableFuture.completedFuture((int) channel.size());
}
public CompletableFuture<Integer> getStartPosition() {
return startPosition;
}
public CompletableFuture<Integer> write(ByteBuffer byteBuffer, CompletableFuture<Integer> currentPosition) {
return currentPosition.thenCompose(position -> {
CompletableFuture<Integer> writenBytes = new CompletableFuture<>();
channel.write(byteBuffer, position, null, new CompletionHandler<Integer, ByteBuffer>() {
@Override
public void completed(Integer result, ByteBuffer attachment) {
writenBytes.complete(result);
}
@Override
public void failed(Throwable exc, ByteBuffer attachment) {
writenBytes.completeExceptionally(exc);
}
});
return writenBytes.thenApply(writenBytesLength -> writenBytesLength + position);
});
}
public void close(CompletableFuture<Integer> currentPosition) {
currentPosition.whenComplete((position, throwable) -> IOUtils.closeQuietly(channel));
}
}
public static void main(String[] args) throws IOException {
final String filepath = "/media/veracrypt4/files/1.jpg";
final String downloadUrl = "https://m0.cl/t/butterfly-3000.jpg";
final AsyncHttpClient client = Dsl.asyncHttpClient(Dsl.config().setFollowRedirect(true)
.setResponseBodyPartFactory(AsyncHttpClientConfig.ResponseBodyPartFactory.LAZY));
final AsynchronousFileChannel channel = AsynchronousFileChannel.open(Paths.get(filepath), StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.CREATE);
final AsyncChannelWriter asyncChannelWriter = new AsyncChannelWriter(channel);
final AtomicReference<CompletableFuture<Integer>> atomicReferencePosition = new AtomicReference<>(asyncChannelWriter.getStartPosition());
client.prepareGet(downloadUrl)
.execute(new AsyncCompletionHandler<Response>() {
@Override
public State onBodyPartReceived(HttpResponseBodyPart content) throws Exception {
//if EAGER, content.getBodyByteBuffer() return HeapByteBuffer, if LAZY, return DirectByteBuffer
final ByteBuffer bodyByteBuffer = content.getBodyByteBuffer();
final CompletableFuture<Integer> currentPosition = atomicReferencePosition.get();
final CompletableFuture<Integer> newPosition = asyncChannelWriter.write(bodyByteBuffer, currentPosition);
atomicReferencePosition.set(newPosition);
return State.CONTINUE;
}
@Override
public Response onCompleted(Response response) {
asyncChannelWriter.close(atomicReferencePosition.get());
return response;
}
});
}
在这种情况下，图片坏了。但是，如果我使用FileChannel而不是AsynchronousFileChannel，则在两种情况下文件都正常输出。使用DirectByteBuffer（如果使用LazyResponseBodyPart.getBodyByteBuffer()）和AsynchronousFileChannel会不会有细微差别？
如果EAGER一切正常，我的代码可能出什么问题了？
我注意到，如果我使用LAZY，例如，我添加以下行
Thread.sleep (10)中的onBodyPartReceived，如下所示：
@Override
public State onBodyPartReceived(HttpResponseBodyPart content) throws Exception {
final ByteBuffer bodyByteBuffer = content.getBodyByteBuffer();
final CompletableFuture<Integer> currentPosition = atomicReferencePosition.get();
final CompletableFuture<Integer> newPosition = finalAsyncChannelWriter.write(bodyByteBuffer, currentPosition);
atomicReferencePosition.set(newPosition);
Thread.sleep(10);
return State.CONTINUE;
}
文件以非损坏状态保存到磁盘。
据我了解，原因是在这10毫秒内，AsynchronousFileChannel中的异步线程设法将数据从该DirectByteBuffer写入磁盘。
事实证明文件已损坏，原因是该异步线程使用此缓冲区与netty线程一起写入。
如果我们用EagerResponseBodyPart查看源代码，那么我们将看到以下内容
private final byte[] bytes;
public EagerResponseBodyPart(ByteBuf buf, boolean last) {
super(last);
bytes = byteBuf2Bytes(buf);
}
@Override
public ByteBuffer getBodyByteBuffer() {
return ByteBuffer.wrap(bytes);
}
因此，当一条数据到达时，它将立即存储在字节数组中。然后，我们可以将它们安全地包装在HeapByteBuffer中，并传输到文件通道中的异步线程。
但是，如果您查看代码LazyResponseBodyPart
private final ByteBuf buf;
public LazyResponseBodyPart(ByteBuf buf, boolean last) {
super(last);
this.buf = buf;
}
@Override
public ByteBuffer getBodyByteBuffer() {
return buf.nioBuffer();
}
如您所见，我们实际上通过方法调用ByteBuff在异步文件通道线程netty PooledSlicedByteBuf（在这种情况下始终为nioBuffer）中使用
在这种情况下，如何安全地在异步线程中传递DirectByteBuffer而不将缓冲区复制到Java堆中？
答案 0 :(得分：0)
我与AsyncHttpClient的维护者进行了交谈。
Can see here
主要问题是我不使用netty ByteBuf方法retain和release。
最后，我提出了两种解决方案。
首先：使用CompletableFuture将字节顺序写入具有跟踪位置的文件。
为AsynchronousFileChannel定义包装类
@Log4j2
public class AsyncChannelNettyByteBufWriter implements Closeable {
private final AtomicReference<CompletableFuture<Long>> positionReference;
private final AsynchronousFileChannel channel;
public AsyncChannelNettyByteBufWriter(AsynchronousFileChannel channel) {
this.channel = channel;
try {
this.positionReference = new AtomicReference<>(CompletableFuture.completedFuture(channel.size()));
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public CompletableFuture<Long> write(ByteBuf byteBuffer) {
final ByteBuf byteBuf = byteBuffer.retain();
return positionReference.updateAndGet(x -> x.thenCompose(position -> {
final CompletableFuture<Integer> writenBytes = new CompletableFuture<>();
channel.write(byteBuf.nioBuffer(), position, byteBuf, new CompletionHandler<Integer, ByteBuf>() {
@Override
public void completed(Integer result, ByteBuf attachment) {
attachment.release();
writenBytes.complete(result);
}
@Override
public void failed(Throwable exc, ByteBuf attachment) {
attachment.release();
log.error(exc);
writenBytes.completeExceptionally(exc);
}
});
return writenBytes.thenApply(writenBytesLength -> writenBytesLength + position);
}));
}
public void close() {
positionReference.updateAndGet(x -> x.whenComplete((position, throwable) -> {
try {
channel.close();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}));
}
}
实际上，这里的记录可能不会出现AtomicReference，如果记录发生在一个线程中，并且如果发生在多个线程中，那么我们就需要认真地进行同步。
主要用法。
public static void main(String[] args) throws IOException {
final String filepath = "1.jpg";
final String downloadUrl = "https://m0.cl/t/butterfly-3000.jpg";
final AsyncHttpClient client = Dsl.asyncHttpClient(Dsl.config().setFollowRedirect(true)
.setResponseBodyPartFactory(AsyncHttpClientConfig.ResponseBodyPartFactory.LAZY));
final AsynchronousFileChannel channel = AsynchronousFileChannel.open(Paths.get(filepath), StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.CREATE);
final AsyncChannelNettyByteBufWriter asyncChannelNettyByteBufWriter = new AsyncChannelNettyByteBufWriter(channel);
client.prepareGet(downloadUrl)
.execute(new AsyncCompletionHandler<Response>() {
@Override
public State onBodyPartReceived(HttpResponseBodyPart content) {
final ByteBuf byteBuf = ((LazyResponseBodyPart) content).getBuf();
asyncChannelNettyByteBufWriter.write(byteBuf);
return State.CONTINUE;
}
@Override
public Response onCompleted(Response response) {
asyncChannelNettyByteBufWriter.close();
return response;
}
});
}
第二种解决方案：根据接收到的字节大小跟踪位置。
public static void main(String[] args) throws IOException {
final String filepath = "1.jpg";
final String downloadUrl = "https://m0.cl/t/butterfly-3000.jpg";
final AsyncHttpClient client = Dsl.asyncHttpClient(Dsl.config().setFollowRedirect(true)
.setResponseBodyPartFactory(AsyncHttpClientConfig.ResponseBodyPartFactory.LAZY));
final ExecutorService executorService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() * 2);
final AsynchronousFileChannel channel = AsynchronousFileChannel.open(Paths.get(filepath), new HashSet<>(Arrays.asList(StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.CREATE)), executorService);
client.prepareGet(downloadUrl)
.execute(new AsyncCompletionHandler<Response>() {
private long position = 0;
@Override
public State onBodyPartReceived(HttpResponseBodyPart content) {
final ByteBuf byteBuf = ((LazyResponseBodyPart) content).getBuf().retain();
long currentPosition = position;
position+=byteBuf.readableBytes();
channel.write(byteBuf.nioBuffer(), currentPosition, byteBuf, new CompletionHandler<Integer, ByteBuf>() {
@Override
public void completed(Integer result, ByteBuf attachment) {
attachment.release();
if(content.isLast()){
try {
channel.close();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
}
@Override
public void failed(Throwable exc, ByteBuf attachment) {
attachment.release();
try {
channel.close();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
});
return State.CONTINUE;
}
@Override
public Response onCompleted(Response response) {
return response;
}
});
}
在第二种解决方案中，由于我们不等到将一些字节写入文件后，AsynchronousFileChannel可以创建很多线程（如果使用Linux，因为Linux不会实现非阻塞异步文件IO。在Windows中，情况要好得多。
正如我的测量结果所示，在写入慢速USB闪存的情况下，线程数可以达到数万，因此，您需要通过创建ExecutorService并传输来限制线程数到AsynchronousFileChannel。
第一种和第二种解决方案是否有明显的优缺点？我很难说。也许有人可以说出什么更有效。