将大数据文件拆分为包含完整行的多个文件

时间:2014-10-08 19:22:37

标签: c#

我想将大数据文件(5 GB)拆分成多个文件(5个1 GB的文件)。

我正在使用此代码: -

string destFileLocation = @"C:\Users\";
int index = 0;
long maxFileSize = 1073741824;
byte[] buffer = new byte[65536];
//int a = buffer.Length;

using (Stream source = File.OpenRead(sourceFileName))
{
    try
    {
        while (source.Position < source.Length)
        {
            index++;
            // Create a new sub File, and read into t
            string newFileName = Path.Combine(destFileLocation, Path.GetFileNameWithoutExtension(sourceFileName));
            //destinationFile = new StreamWriter(
            //        string.Format(destinationFileName, fileCounter + 1));
            newFileName += "_" + index.ToString() + Path.GetExtension(sourceFileName);

            using (Stream destination = File.OpenWrite(newFileName))
            {
                try
                {
                    while (destination.Position < maxFileSize)
                    {
                         int bytes = source.Read(buffer, 0, (int)Math.Min(maxFileSize, buffer.Length));
                         destination.Write(buffer, 0, bytes);

                         if (bytes < Math.Min(maxFileSize, buffer.Length))
                         {
                             break;
                         }
                    }
                }
                finally
                {
                    destination.Dispose();
                    destination.Close();
                }
            }
        }
    }
    finally
    {
        source.Dispose();
        source.Close();
    }
}

现在文件在行之间分割,但我们需要完整的行。

请提供一些建议。

0 个答案:

没有答案