Question

我最初在GitHub上提出了这个问题，但是没有人回复，所以我在这里尝试。

我们正在使用Azure Storage Data Movement Library上传大文件，并希望使用 checkpoints 支持暂停/恢复。但是上传没有完成并且卡在while循环中：

TransferManager.Configurations.BlockSize = 4 * 1024 * 1024; // 4 MB

// create stream with 100 MB
var content = Enumerable
    .Range(0, 100 * 1024 * 1024)
    .Select(i => (byte)'b')
    .ToArray();
var source = new MemoryStream(content);

TransferCheckpoint checkPoint = null;
while (true)
{
    Console.Write(".");
    // Resume if there is already a checkpoint
    var context = checkPoint != null
        ? new SingleTransferContext(checkPoint)
        : new SingleTransferContext();

    context.ShouldOverwriteCallback = (src, dest) => true;
    context.LogLevel = Microsoft.WindowsAzure.Storage.LogLevel.Verbose;
    var recorder = new ProgressRecorder();
    context.ProgressHandler = recorder;
    try
    {
        // Interrupt upload after 20 seconds
        using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(20)))
        {
            await TransferManager.UploadAsync(source, blob, null, context, cts.Token);
        }
        // break if no exception occured, i.e. the upload finished
        break;
    }
    catch (OperationCanceledException) { }
    checkPoint = context.LastCheckpoint; // save the last checkpoint
}

我们在保存检查点后添加了一些调试输出，即我们使用Storage API获取所有未提交的块并将其与recorder匹配。

我们得到以下输出：

Uncommitted blocks according to Storage API
A6689033-D96E-4F9B-B238-E27891C1B1FC/custom-upload-test3: 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 = 33554432 (32 MB)
Transfer statistics according to Data Movement Library
Transferred bytes: 33554432 (32 MB); Transfered: 0; Skipped: 0, Failed: 1

Uncommitted blocks according to Storage API
A6689033-D96E-4F9B-B238-E27891C1B1FC/custom-upload-test3: 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 = 46137344 (44 MB)
Transfer statistics according to Data Movement Library
Transferred bytes: 79691776 (76 MB); Transfered: 0; Skipped: 0, Failed: 2

Uncommitted blocks according to Storage API
A6689033-D96E-4F9B-B238-E27891C1B1FC/custom-upload-test3: 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 = 46137344 (44 MB)
Transfer statistics according to Data Movement Library
Transferred bytes: 121634816 (116 MB); Transfered: 0; Skipped: 0, Failed: 3

Uncommitted blocks according to Storage API
A6689033-D96E-4F9B-B238-E27891C1B1FC/custom-upload-test3: 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 = 46137344 (44 MB)
Transfer statistics according to Data Movement Library
Transferred bytes: 163577856 (156 MB); Transfered: 0; Skipped: 0, Failed: 4

我们使用以下代码段来获取未提交的块列表：

var blockList = blob.DownloadBlockList(BlockListingFilter.Uncommitted);
var expr = string.Join(" + ", blockList.Select(b => b.Length));
var size = blockList.Sum(b => b.Length);
Console.WriteLine($"{blob.Name}: {expr} = {size} ({size/1024.0/1024.0} MB)");

因此，我们似乎没有正确捕获上下文，因为在每次迭代时上传似乎都会重新启动。

我们做错了什么？

使用检查点以块的形式上传大文件

0 个答案: