如何减少大量文件传输的内存使用?

时间:2015-04-27 20:26:05

标签: web-services file-io file-transfer webservice-client

我必须将一些大文件(2GB-ish)传输到Web服务:

public bool UploadContent(System.Web.HttpContext context)
{
     var file = context.Request.Files[0];
     var fileName = file.FileName;

     byte[] fileBytes = new Byte[file.ContentLength];
     file.InputStream.Read(fileBytes, 0, fileBytes.Length);
     client.createResource(fileBytes);
}        

HttpContext已经包含File[0]中文件的内容,但是我无法看到将这些字节传递给Web服务的createResource(byte[] contents)方法的方法将副本制作为byte数组...所以我正在吃糖果等记忆。

有更有效的方法吗?

编辑 client.createResource()是COTS产品的一部分,修改不在我们的控制范围内。

1 个答案:

答案 0 :(得分:0)

您可以发送文件块,而不是发送整个字节。寻找文件一步一步上传并合并下一个块以便在服务器上保存字节。 只有在您允许修改该方法时才需要更新client.CreateResource方法:)

添加以下参数: string fileName //在开始发送块时找到文件名 byte [] buffer //将通过webservice发送到服务器的块 长偏移量//将告诉您已上载了多少数据的信息,以便您可以搜索文件并合并缓冲区。

现在您的方法将如下所示:

public bool CreateResource(string FileName, byte[] buffer, long Offset)
{
    bool retVal = false;
    try
    {
        string FilePath = "d:\\temp\\uploadTest.extension";
        if (Offset == 0)
            File.Create(FilePath).Close();
        // open a file stream and write the buffer. 
        // Don't open with FileMode.Append because the transfer may wish to 
        // start a different point
        using (FileStream fs = new FileStream(FilePath, FileMode.Open,
            FileAccess.ReadWrite, FileShare.Read))
        {
            fs.Seek(Offset, SeekOrigin.Begin);
            fs.Write(buffer, 0, buffer.Length);
        }
        retVal = true;
    }
    catch (Exception ex)
    {
        // Log exception or send error message to someone who cares
    }
    return retVal;
}

现在从HttpPostedFile的InputStream中读取块中的文件,尝试下面的代码:

public bool UploadContent(System.Web.HttpContext context)
{
    //the file that we want to upload
    var file = context.Request.Files[0];
    var fs = file.InputStream;

    int Offset = 0; // starting offset.

    //define the chunk size
    int ChunkSize = 65536; // 64 * 1024 kb

    //define the buffer array according to the chunksize.
    byte[] Buffer = new byte[ChunkSize];
    //opening the file for read.

    try
    {
        long FileSize = file.ContentLength; // File size of file being uploaded.
        // reading the file.
        fs.Position = Offset;
        int BytesRead = 0;
        while (Offset != FileSize) // continue uploading the file chunks until offset = file size.
        {
            BytesRead = fs.Read(Buffer, 0, ChunkSize); // read the next chunk 

            if (BytesRead != Buffer.Length)
            {
                ChunkSize = BytesRead;
                byte[] TrimmedBuffer = new byte[BytesRead];
                Array.Copy(Buffer, TrimmedBuffer, BytesRead);
                Buffer = TrimmedBuffer; // the trimmed buffer should become the new 'buffer'
            }
            // send this chunk to the server. it is sent as a byte[] parameter, 
            // but the client and server have been configured to encode byte[] using MTOM. 
            bool ChunkAppened = client.createResource(file.FileName, Buffer, Offset);
            if (!ChunkAppened)
            {
                break;
            }

            // Offset is only updated AFTER a successful send of the bytes. 
            Offset += BytesRead; // save the offset position for resume
        }
    }
    catch (Exception ex)
    {
    }
    finally
    {
        fs.Close();
    }
}
  

免责声明:我还没有测试过这段代码。这是一个示例代码,用于显示如何在不妨碍内存的情况下实现大型文件上载。

参考:Source article.