Question

以下是从Google云端存储下载文件的代码：

@Override
public void write(OutputStream outputStream) throws IOException {
    try {
        LOG.info(path);
        InputStream stream = new ByteArrayInputStream(GoogleJsonKey.JSON_KEY.getBytes(StandardCharsets.UTF_8));
        StorageOptions options = StorageOptions.newBuilder()
                .setProjectId(PROJECT_ID)
                .setCredentials(GoogleCredentials.fromStream(stream)).build();
        Storage storage = options.getService();
        final CountingOutputStream countingOutputStream = new CountingOutputStream(outputStream);
        byte[] read = storage.readAllBytes(BlobId.of(BUCKET, path));
        countingOutputStream.write(read);
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        outputStream.close();
    }
}

这样可行，但问题在于它必须首先缓冲所有字节，然后再将其流回此方法的客户端。这导致了很多延迟，尤其是当存储在GCS中的文件很大时。

有没有办法从GCS获取文件，直接将其流式传输到OutputStream ，这个这个OutputStream btw用于Servlet。

Answer 1

为了澄清，您需要OutputStream还是InputStream？一种看待这种情况的方法是将存储在Google云端存储对象中的数据作为文件存储，并且您有一个InputStream来读取该文件。如果可行，请继续阅读。

Storage API中没有提供InputStream或OutputStream的现有方法。但是有2 APIs in the Cloud Storage client library公开ReadChannel对象，该对象从ReadableByteChannel扩展而来（来自java NIO API）。

ReadChannel reader(String bucket, String blob, BlobSourceOption... options);
ReadChannel reader(BlobId blob, BlobSourceOption... options);

使用此示例的简单示例（取自StorageSnippets.java）：

/**
   * Example of reading a blob's content through a reader.
   */
  // [TARGET reader(String, String, BlobSourceOption...)]
  // [VARIABLE "my_unique_bucket"]
  // [VARIABLE "my_blob_name"]
  public void readerFromStrings(String bucketName, String blobName) throws IOException {
    // [START readerFromStrings]
    try (ReadChannel reader = storage.reader(bucketName, blobName)) {
      ByteBuffer bytes = ByteBuffer.allocate(64 * 1024);
      while (reader.read(bytes) > 0) {
        bytes.flip();
        // do something with bytes
        bytes.clear();
      }
    }
    // [END readerFromStrings]
  }

您还可以使用newInputStream()方法将InputStream打包到ReadableByteChannel上。

public static InputStream newInputStream(ReadableByteChannel ch)

即使您需要OutputStream，您也应该能够将InputStream或更好的数据从ReadChannel对象复制到OutputStream。

完整示例

将此示例运行为：PROGRAM_NAME <BUCKET_NAME> <BLOB_PATH>

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.nio.channels.WritableByteChannel;

import com.google.cloud.ReadChannel;
import com.google.cloud.storage.Bucket;
import com.google.cloud.storage.BucketInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

/**
 * An example which reads the contents of the specified object/blob from GCS
 * and prints the contents to STDOUT.
 *
 * Run it as PROGRAM_NAME <BUCKET_NAME> <BLOB_PATH>
 */
public class ReadObjectSample {
  private static final int BUFFER_SIZE = 64 * 1024;

  public static void main(String[] args) throws IOException {
    // Instantiates a Storage client
    Storage storage = StorageOptions.getDefaultInstance().getService();

    // The name for the GCS bucket
    String bucketName = args[0];
    // The path of the blob (i.e. GCS object) within the GCS bucket.
    String blobPath = args[1];

    printBlob(storage, bucketName, blobPath);
  }

  // Reads from the specified blob present in the GCS bucket and prints the contents to STDOUT.
  private static void printBlob(Storage storage, String bucketName, String blobPath) throws IOException {
    try (ReadChannel reader = storage.reader(bucketName, blobPath)) {
      WritableByteChannel outChannel = Channels.newChannel(System.out);
      ByteBuffer bytes = ByteBuffer.allocate(BUFFER_SIZE);
      while (reader.read(bytes) > 0) {
        bytes.flip();
        outChannel.write(bytes);
        bytes.clear();
      }
    }
  }
}

Answer 2

目前我能找到的最干净的选项如下：

Blob blob = bucket.get("some-file");
ReadChannel reader = blob.reader();
InputStream inputStream = Channels.newInputStream(reader);

Channels来自java.nio。此外，您还可以使用commons io轻松将InputStream读入OutputStream：

IOUtils.copy(inputStream, outputStream);

Answer 3

代码，基于@Tuxdude答案

 @Nullable
    public byte[] getFileBytes(String gcsUri) throws IOException {

        Blob blob = getBlob(gcsUri);
        ReadChannel reader;
        byte[] result = null;
        if (blob != null) {
            reader = blob.reader();
            InputStream inputStream = Channels.newInputStream(reader);
           result = IOUtils.toByteArray(inputStream);
        }
        return result;
    }

或

//this will work only with files 64 * 1024 bytes on smaller
 @Nullable
    public byte[] getFileBytes(String gcsUri) throws IOException {
        Blob blob = getBlob(gcsUri);

        ReadChannel reader;
        byte[] result = null;
        if (blob != null) {
            reader = blob.reader();
            ByteBuffer bytes = ByteBuffer.allocate(64 * 1024);

            while (reader.read(bytes) > 0) {
                bytes.flip();
                result = bytes.array();
                bytes.clear();
            }
        }
        return result; 
    }

帮助程序代码：

   @Nullable
    Blob getBlob(String gcsUri) {
        //gcsUri is "gs://" + blob.getBucket() + "/" + blob.getName(),
        //example "gs://myapp.appspot.com/ocr_request_images/000c121b-357d-4ac0-a3f2-24e0f6d5cea185dffb40eee-850fab211438.jpg"

        String bucketName = parseGcsUriForBucketName(gcsUri);
        String fileName = parseGcsUriForFilename(gcsUri);

        if (bucketName != null && fileName != null) {
            return storage.get(BlobId.of(bucketName, fileName));
        } else {
            return null;
        }
    }

    @Nullable
    String parseGcsUriForFilename(String gcsUri) {
        String fileName = null;
        String prefix = "gs://";
        if (gcsUri.startsWith(prefix)) {
            int startIndexForBucket = gcsUri.indexOf(prefix) + prefix.length() + 1;
            int startIndex = gcsUri.indexOf("/", startIndexForBucket) + 1;
            fileName = gcsUri.substring(startIndex);
        }
        return fileName;
    }

    @Nullable
    String parseGcsUriForBucketName(String gcsUri) {
        String bucketName = null;
        String prefix = "gs://";
        if (gcsUri.startsWith(prefix)) {
            int startIndex = gcsUri.indexOf(prefix) + prefix.length();
            int endIndex = gcsUri.indexOf("/", startIndex);
            bucketName = gcsUri.substring(startIndex, endIndex);
        }
        return bucketName;
    }

Answer 4

使用google-cloud-nio通过Google Cloud Storage流式传输文件的另一种（便捷）方式：

Path path = Paths.get(URI.create("gs://bucket/file.csv"));
InputStream in = Files.newInputStream(path);

来自Google云端存储的流文件

4 个答案:

完整示例