Question

从MongoDB 2 Azure Blob存储迁移文件时遇到问题。

下一个方法获取GridFSFile对象（代表MongoDB GridFSFileStorage中的文件），然后调用uploadMemoryStream方法进行上传。

值得一提的是，gridFSFile在findById之后确实包含内容，并且长度dows也包含内容，并且位置最初是0。

gridFSFile.Open方法创建一个Stream对象，然后将其作为参数传递给上载。

private static void iterateOverVersionCollection(Version version, Asset asset)
{
    try
    {    
        string _gridFSId = version.GridFSId;
        GridFSFile gridFSFile = gridFSFileStorage.FindById(_gridFSId);
        if (gridFSFile == null) return;

        string size = version.Name.ToLower();
        asset.Size = size;
        CloudBlockBlob blockBlob = GetBlockBlobReference(version, gridFSFile, asset);
        uploadMemoryStream(blockBlob, gridFSFile, asset);
        asset.UploadedOK = true;
    }
    catch (StorageException ex)
    {
        asset.UploadedOK = false;
        logException(ex, asset);
    }
}

private static void uploadMemoryStream(CloudBlockBlob blockBlob, GridFSFile gridFSFile, Asset asset)
{
      Stream st = gridFSFile.Open();
      blockBlob.UploadFromStream(st);
}

UploadFromStream永远占用并且永远不进行上传，而且要提到的一件事是，无论我如何使用gridFSFile，如果我尝试使用Stream.copyTo c＃方法用它创建一个MemoryStream，它也将永远永无止境因此应用程序被卡在blockBlob.UploadFromStream（st）;

我不仅尝试将gridFSFile.Open传递给UploadFromMemoryStream，还尝试了下一段代码：

using (var stream = new MemoryStream())
{
    byte[] buffer = new byte[2048]; // read in chunks of 2KB
    int bytesRead;
    while((bytesRead = st.Read(buffer, 0, buffer.Length)) > 0)
    {
        stream.Write(buffer, 0, bytesRead);
    }
    byte[] result = stream.ToArray();
}

但还是一样，程序卡在st.Read行中。

任何帮助将不胜感激。

Answer 1

请注意，由于UploadFromFileAsync（）或UploadFromStream对于庞大的Blob而言不是可靠且高效的操作，因此建议您考虑以下替代方案：

如果可以接受命令行工具，则可以尝试AzCopy，它可以高性能传输Azure存储数据，并且可以暂停和恢复其传输。

如果要以编程方式控制传输作业，请使用Azure Storage Data Movement Library，它是AzCopy的核心。相同代码的示例代码

string storageConnectionString = "myStorageConnectionString";
     CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
      CloudBlobClient blobClient = account.CreateCloudBlobClient();
      CloudBlobContainer blobContainer = blobClient.GetContainerReference("mycontainer");
      blobContainer.CreateIfNotExistsAsync().Wait();
      string sourcePath = @"C:\Tom\TestLargeFile.zip";
      CloudBlockBlob destBlob = blobContainer.GetBlockBlobReference("LargeFile.zip");
      // Setup the number of the concurrent operations
      TransferManager.Configurations.ParallelOperations = 64;
      // Setup the transfer context and track the upoload progress
      var context = new SingleTransferContext
      {
          ProgressHandler =
          new Progress<TransferStatus>(
               progress => { Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred); })
       };
      // Upload a local blob
      TransferManager.UploadAsync(sourcePath, destBlob, null, context, CancellationToken.None).Wait();
      Console.WriteLine("Upload finished !");
      Console.ReadKey();

如果您仍然希望通过编程方式从流中上传文件，我建议您分块上传，可以使用下面的代码

var container = _client.GetContainerReference("test");
container.CreateIfNotExists();
var blob = container.GetBlockBlobReference(file.FileName);
var blockDataList = new Dictionary<string, byte[]>();
using (var stream = file.InputStream)
{
    var blockSizeInKB = 1024;
    var offset = 0;
    var index = 0;
    while (offset < stream.Length)
    {
        var readLength = Math.Min(1024 * blockSizeInKB, (int)stream.Length - offset);
        var blockData = new byte[readLength];
        offset += stream.Read(blockData, 0, readLength);
        blockDataList.Add(Convert.ToBase64String(BitConverter.GetBytes(index)), blockData);

        index++;
    }
}

Parallel.ForEach(blockDataList, (bi) =>
{
    blob.PutBlock(bi.Key, new MemoryStream(bi.Value), null);
});
blob.PutBlockList(blockDataList.Select(b => b.Key).ToArray());

另一方面，如果您的系统中有文件可用，并且想使用Uploadfile方法，那么我们也可以灵活地使用该方法来上传文件数据（大块）

TimeSpan backOffPeriod = TimeSpan.FromSeconds(2);
int retryCount = 1;
BlobRequestOptions bro = new BlobRequestOptions()
{
  SingleBlobUploadThresholdInBytes = 1024 * 1024, //1MB, the minimum
  ParallelOperationThreadCount = 1, 
  RetryPolicy = new ExponentialRetry(backOffPeriod, retryCount),
};
CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(ConnectionString);
CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
cloudBlobClient.DefaultRequestOptions = bro;
cloudBlobContainer = cloudBlobClient.GetContainerReference(ContainerName);
CloudBlockBlob blob = cloudBlobContainer.GetBlockBlobReference(Path.GetFileName(fileName));
blob.StreamWriteSizeInBytes = 256 * 1024; //256 k
blob.UploadFromFile(fileName, FileMode.Open);

有关详细说明，请浏览

https://www.red-gate.com/simple-talk/cloud/platform-as-a-service/azure-blob-storage-part-4-uploading-large-blobs/

希望有帮助。

C＃使用“ UploadFromStream”方法将文件从MongoDB迁移到Azure Blob存储不起作用

1 个答案: