在构建Zip文件时减少内存使用量

时间:2013-03-12 13:17:50

标签: .net zipfile

我的应用程序具有从数据库导出数据的功能。它使用Ionic ZIP库构建一个包含数据的zip文件。

该过程通过以每行1,000行的页面检索数据来工作。检索的数据包含每行数据的2个图像(JPEGS)。 “导出”包含一个HTML文件,其中包含HTML表格。该表包含每行的重要信息和图像链接。每个图像都作为JPEG文件存储在ZIP文件中。编写的HTML包含一个锚标记,链接到相应的图像文件。

一切正常,但我最近发现在导出大量行时会抛出OutOfMemoryException(在我发现问题的情况下超过24,000行)。在查看我的代码时,我没有看到任何明显的泄漏(或者我不会发布这个,是吗?;))

以下是输出正在导出的数据的方法:

private void OutputReads( StreamWriter writer, ZipFile zipFile, BackgroundWorker worker, ExportArgs args ) {
    int ReadCount = Math.Min( args.ReadMatches, args.ResultsLimit );

    writer.WriteLine( "<h2>Matching Reads:</h2>" );
    writer.WriteLine( "<p>Number in Database: {0}</p>", args.ReadMatches.ToString( "#,##0" ) );
    writer.WriteLine( "<p>Number in Report:   {0}</p>",      ReadCount  .ToString( "#,##0" ) );
    writer.WriteLine();

    if ( args.ReadMatches == 0 ) {
        return;
    }

    writer.WriteLine( "<table border=\"1\" cellspacing=\"0\" bordercolordark=\"black\" bordercolorlight=\"black\">" );
    writer.WriteLine( "<tr>" );
    writer.WriteLine( "<th width=\"110\"><b>Plate</b></th>" );
    writer.WriteLine( "<th width=\"60\"><b>State</b></th>" );
    writer.WriteLine( "<th width=\"200\"><b>Date / Time</b></th>" );
    writer.WriteLine( "<th width=\"142\"><b>Latitude</b?</th> " );
    writer.WriteLine( "<th width=\"142\"><b>Longitude</b?</th> " );
    writer.WriteLine( "<th width=\"125\"><b>Alarm Class</b></th>" );
    writer.WriteLine( "<th width=\"150\"><b>Notes</b></th>" );
    writer.WriteLine( "<th width=\"150\"><b>Officer Notes</b></th>" );
    writer.WriteLine( "<th width=\"150\"><b>Images</b></th>" );
    writer.WriteLine( "</tr>" );

    int noPages = ReadCount / args.PageSize + ( ReadCount % args.PageSize == 0 ? 0 : 1 );

    for ( int pageNo = 0, count = 0; pageNo < noPages; pageNo++ ) {
        if ( worker.CancellationPending ) {
            return;
        }

        ReadViewModel[] reads = null;

        try {
            reads = DataInterface.GetReads( args.LocaleCode, args.Plate, 
                                            args.StartDate, args.EndDate, 
                                            args.AlarmClasses, args.HotListId, 
                                            args.PageSize, pageNo
                                          );

        } catch ( DataAccessException ex ) {
            . . .
        } catch ( ThreadAbortException ) {
            // The thread is stopping.  Stop processing now.

        } catch ( Exception ex ) {
            . . .
        }

        foreach ( ReadViewModel read in reads ) {
            if ( worker.CancellationPending ) {
                return;
            }

            writer.WriteLine( "<tr>" );
            writer.WriteLine( "<td width=\"110\"><p align=\"left\" style=\"text-align:left\">{0}</p></td>"    , read.Plate );
            writer.WriteLine( "<td width=\"60\" ><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.State );
            writer.WriteLine( "<td width=\"150\"><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.TimeStamp );
            writer.WriteLine( "<td width=\"125\"><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.GPSInformation == null ? "&nbsp;" : read.GPSInformation.Position.Latitude.ToString(  "##0.000000" ) );
            writer.WriteLine( "<td width=\"125\"><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.GPSInformation == null ? "&nbsp;" : read.GPSInformation.Position.Longitude.ToString( "##0.000000" ) );
            writer.WriteLine( "<td width=\"125\"><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.AlarmClass     == null ? "&nbsp;" : read.AlarmClass );
            writer.WriteLine( "<td width=\"150\"><p align=\"left\" style=\"text-align:left\">{0}</p></td>"    , read.       Notes   == null ? "&nbsp;" : read.       Notes );
            writer.WriteLine( "<td width=\"150\"><p align=\"left\" style=\"text-align:left\">{0}</p></td>"    , read.OfficerNotes   == null ? "&nbsp;" : read.OfficerNotes );

            if ( read.ImageData == null ) {
                try {
                    DataInterface.GetPlateImage( read );
                } catch ( DataAccessException ex ) {
                    . . .
                } catch ( Exception ex ) {
                    . . .
                }
            }

            if ( read.OverviewImages == null ) {
                try {
                    DataInterface.GetOverviewImages( read );
                } catch ( DataAccessException ex ) {
                    . . .
                } catch ( Exception ex ) {
                    . . .
                }
            }

            if ( read.ImageData != null ) {
                string ext = LPRCore.CarSystem.ImageDataAccessor.GetImageFileExtension( read.ImageData );
                writer.Write( "<td width=\"150\"><p align=\"left\" style=\"text-align:left\"><a href=\".\\Images\\{0}.{1}\" target=\"_blank\">BW</a>", read.ID.ToString( "N" ), ext );

                string fileName = string.Format( ".\\Images\\{0}{1}", read.ID.ToString( "N" ), ext );

                if ( !zipFile.ContainsEntry( fileName ) ) {
                    zipFile.AddEntry( fileName, read.ImageData );
                }
            } else {
                writer.Write( "No Plate Image" );
            }

            if ( read.OverviewImages != null && read.OverviewImages.Length > 0 ) {
                for ( int i = 0; i < read.OverviewImages.Length; i++ ) {
                    string ext = LPRCore.CarSystem.ImageDataAccessor.GetImageFileExtension( read.OverviewImages[ i ].ImageData );
                    writer.Write( " - <a href=\".\\Images\\{0}_C{1}{2}\" target=\"_blank\">Color {1}</a>", read.ID.ToString( "N" ), i == 0 ? string.Empty : i.ToString(), ext );

                    string fileName = string.Format( ".\\Images\\{0}_c{1}{2}", read.ID.ToString( "N" ), i == 0 ? string.Empty : i.ToString(), ext );
                    if ( !zipFile.ContainsEntry( fileName ) ) {
                        zipFile.AddEntry( fileName, read.OverviewImages[ i ].ImageData );
                    }
                }
            } else {
                writer.Write( "No Overview Images" );
            }

            writer.WriteLine( "</p></td>" );
            writer.WriteLine( "</tr>" );
            count++;
            worker.ReportProgress( count, args );
        }
    }
    writer.WriteLine( "</table>" );
    writer.WriteLine();
}

我认为Zip文件可能是问题,因为它没有刷新到磁盘或类似的东西,只是在处理行时不断变大和变大。

有没有办法可以将zip文件刷新到磁盘并释放所有图像,以便垃圾收集器释放它们,还是有另一种方法使用这个库来使用更少的内存来构建zip文件?

1 个答案:

答案 0 :(得分:0)

我能够在另一位开发人员的帮助下为此解决这个问题。

DotNetZip库定义ZipFile.AddEntry方法的重载,该方法将类型为WriteDelegate的委托作为参数。委托传递文件的名称和Stream,文件的内容必须作为参数写入。{/ p>

我在我的代码中定义了一个名为GetImageBytes的方法:

private void GetImageBytes( string entryName, Stream stream ) {
    Guid imageId;
    Guid.TryParse( Path.GetFileName( entryName ).Substring( 0, 32 ), out imageId );
    using ( BinaryWriter writer = new BinaryWriter( stream ) ) {
        try {
            writer.Write( DataInterface.GetImageData( imageId ) );

        } catch ( DataAccessException ex ) {
            DbMonitor.HandleDatabaseError( ex );

        } catch ( Exception ex ) {
            . . .
        }
    }
}

为了完成这项工作,我必须在文件名中对数据库中的图像ID进行编码。代码从文件名解析ID,然后通过调用Service层中的方法从数据库中检索图像字节。最后,它会创建BinaryWriter并将图像数据发送到Stream

我的程序现在可以导出超过24,000行数据而不会抛出OutOfMemoryException