我正在尝试使用HttpWebRequest下载100GB文件。下载将根据预设的部件尺寸拆分为多个部分。以下是我用来下载文件的代码：
private static void AddRangeHeaderHack(WebHeaderCollection headers, long start, long end)
{
// Original workaround by Eric Cadwell, code taken from
// https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=93714
Type type = headers.GetType();
System.Reflection.MethodInfo setAddVerified = type.GetMethod("SetAddVerified",
System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.FlattenHierarchy
);
string rangeHeaderValue = String.Format("bytes={0}-{1}", start, end);
if (setAddVerified != null)
setAddVerified.Invoke(headers, new object[] { "Range", rangeHeaderValue });
}
private ulong GetRemoteFileSize(string URI)
{
ulong size = 0;
HttpWebRequest req = null;
try
{
req = (HttpWebRequest)WebRequest.Create(URI);
using (var res = (HttpWebResponse)req.GetResponse())
{
size = (ulong)res.ContentLength;
res.Close();
}
}
catch (Exception ex)
{
}
if (req != null)
{
try
{
req.Abort();
req = null;
}
catch (Exception)
{
}
}
return size;
}
private int DownloadFromLink(string sSource, string sDestination)
{
int nRetryCount = 0;
int nMaxRetry = 5;
var lastProgress = DateTime.Now;
ulong offset = 0;
var bRetrying = false;
var bResumable = false;
var fileSize = GetRemoteFileSize(sSource);
if (fileSize > 0)
bResumable = true;
while (true)
{
HttpWebRequest webRequest = null;
try
{
try
{
bRetrying = false;
do
{
try
{
if (bDownloadAbort)
{
return -1;
}
webRequest = (HttpWebRequest)WebRequest.Create(sSource);
webRequest.Timeout = 3600000;
if (offset > 0)
{
AddRangeHeaderHack(webRequest.Headers, (long)offset, (long)fileSize);
}
// Retrieve the response from the server
using (var webResponse = (HttpWebResponse)webRequest.GetResponse())
{
var acceptRanges = String.Compare(webResponse.Headers["Accept-Ranges"], "bytes", true) == 0;
// Open the URL for download
using (var streamResponse = webResponse.GetResponseStream())
{
if (streamResponse != null)
{
// Create a new file stream where we will be saving the data (local drive)
using (var streamLocal = new FileStream(sDestination, offset>0?FileMode.Append:FileMode.Create, FileAccess.Write, FileShare.ReadWrite))
{
// It will store the current number of bytes we retrieved from the server
int bytesSize = 0;
// A buffer for storing and writing the data retrieved from the server
byte[] downBuffer = new byte[/*16384*/ 1024 * 1024];
bool binitialtry = true;
int nRetries = 0;
if (offset > 0)
{
streamLocal.Seek((long)offset, SeekOrigin.Begin);
}
// Loop through the buffer until the buffer is empty
while ((bytesSize = streamResponse.Read(downBuffer, 0, downBuffer.Length)) > 0
|| (File.Exists(sDestination) && (offset < (ulong)fileSize) && nRetries < 5 && bResumable))
{
if (binitialtry && bytesSize == 0)
{
binitialtry = false;
}
if (!binitialtry && bytesSize == 0)
{
nRetries++;
bRetrying = nRetries<5;
break;
}
if (bDownloadAbort)
{
try { streamLocal.Close(); }
catch { }
return;
}
try
{
// Write the data from the buffer to the local hard drive
streamLocal.Write(downBuffer, 0, bytesSize);
offset += (ulong)bytesSize;
}
catch (IOException ex)
{
if (streamResponse != null)
streamResponse.Close();
if (streamLocal != null)
streamLocal.Close();
if (webRequest != null)
webRequest.Abort();
return -1;
}
Interlocked.Add(ref actualDownloaded, bytesSize);
}
// When the above code has ended, close the streams
if (streamResponse != null)
streamResponse.Close();
if (streamLocal != null)
try { streamLocal.Close(); }
catch { }
if (webRequest != null)
webRequest.Abort();
if (webRequest != null)
wcDownload.Dispose();
streamLocal.Close();
}
streamResponse.Close();
}
}
webResponse.Close();
}
if(!bRetrying)
break;
}
catch (IOException ex)
{
if (webRequest != null)
webRequest.Abort();
if (wcDownload != null)
wcDownload.Dispose();
if (nRetryCount <= nMaxRetry)
{
Thread.Sleep(10000);
nRetryCount++;
bRetrying = true;
}
else
{
break;
}
}
catch (UnauthorizedAccessException ex)
{
if (webRequest != null)
webRequest.Abort();
break;
}
catch (WebException ex)
{
if (webRequest != null)
webRequest.Abort();
if (wcDownload != null)
wcDownload.Dispose();
if (nRetryCount <= nMaxRetry)
{
Thread.Sleep(10000);
nRetryCount++;
bRetrying = true;
}
else
{
break;
}
}
finally
{
}
} while (bRetrying);
}
catch (Exception ex)
{
break;
}
}
catch
{
break;
}
if(!bRetrying)
break;
}
}
如果我尝试以1个部分下载文件，而不添加范围标题，则代码运行顺畅，文件正常下载。当我添加一个范围标题，比如从10GB到15GB或坦率地说任何值时，代码到达streamResponse.Read并在那里挂起几分钟，然后抛出异常：
无法从传输连接中读取数据：现有数据 连接被远程主机强行关闭
当代码在异常后重试连接时，下载正常恢复，客户端能够从流中读取数据。
有人可以帮我确定为什么会这样吗？
为了清楚有关服务器的问题，该文件当前托管在Amazon S3服务器上，下载是通过生成的直接链接完成的。
答案 0 :(得分：0)
根据http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1.4
，它可能是服务器设置使用持久连接的客户端应该限制它们维护到给定服务器的同时连接数。单用户客户端不应该与任何服务器或代理保持2个以上的连接。代理应该使用最多2 * N个连接到另一个服务器或代理，其中N是同时活动用户的数量。这些指南旨在改善HTTP响应时间并避免拥塞。
尝试使用FDM，看看是否有问题。 http://www.freedownloadmanager.org/