Question

我通过获取数据表然后循环遍历该数据表并写入CSV文件的每一行来创建CSV文件。我的数据源通常有大约65,000行。此过程需要几分钟才能从浏览器下载。问题是本地和开发人员不需要太长时间，但在客户端他们通常会超时。

有更快的方法来生成此文件吗？

Function GenerateCSVFile() As String
    Dim stuPro As New studentProvider.StudentProvider
    Dim emailCenter As New EmailCenter
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "")
    Dim strResult As String = ""

    Dim dtStudent As Data.DataTable
    Dim paymentYear As String = ""
    dtStudent = stuPro.generateDataFile()

    If dtStudent.Rows.Count > 0 Then

        Using sw As New System.IO.StreamWriter(Server.MapPath("Temp/" + strFileName + ".csv"))
            Try
                Dim lineValue As String = ""

                lineValue += "Academic Year, StudentID, SSN, First, Middle, Last"

                sw.WriteLine(lineValue)

                For i As Integer = 0 To dtStudent.Rows.Count - 1

                    lineValue = dtStudent.Rows(i)("fy").ToString
                    lineValue += "," & dtStudent.Rows(i)("uniq_stu_id").ToString
                    lineValue += "," & dtStudent.Rows(i)("ssn").ToString
                    lineValue += "," & dtStudent.Rows(i)("fname").ToString
                    lineValue += "," & dtStudent.Rows(i)("mname").ToString
                    lineValue += "," & dtStudent.Rows(i)("lname").ToString
                    sw.WriteLine(lineValue)

                Next
            Catch ex As Exception
                strResult += ex.ToString
            Finally
                sw.Close()
            End Try

        End Using

        Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv"

        If String.IsNullOrEmpty(strResult) Then

            Dim fs As System.IO.FileStream = Nothing

            fs = System.IO.File.Open(Server.MapPath("Temp/" + strFileName + ".csv"), System.IO.FileMode.Open)
            Dim btFile(fs.Length) As Byte
            fs.Read(btFile, 0, fs.Length)
            fs.Close()

            With Response
                .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName)
                .ContentType = "application/octet-stream"
                .BinaryWrite(btFile)
                .End()
            End With
        End If
    Else
        strResult = "No records found for specified academic year"
    End If

    Return strResult
End Function

更新代码

Function GenerateCSVFile() As String
    Dim startDate As Date = Date.Now
    Dim enddate As Date = Nothing
    Dim stuPro As New studentProvider.StudentProvider
    Dim emailCenter As New EmailCenter
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "")
    Dim strResult As String = ""

    Dim dtStudent As Data.DataTable
    Dim paymentYear As String = ""
    dtStudent = stuPro.generateDataFile(Session("VendorID"), txtAcademicYear.Text.Trim)

    If dtStudent.Rows.Count > 0 Then

        With Response

            Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv"
            .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName)
            .ContentType = "application/octet-stream"

            Dim lineValue As StringBuilder = New StringBuilder

            lineValue.Append("Academic Year, StudentID, SSN, First, Middle, Last")

            .Write(lineValue.ToString)

            For i As Integer = 0 To dtStudent.Rows.Count - 1

                lineValue = New StringBuilder
                lineValue.Append(dtStudent.Rows(i)("fy").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("uniq_stu_id").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("ssn").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("fname").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("mname").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("lname").ToString)

                .Write(lineValue.ToString)

            Next
            enddate = Date.Now

            MsgBox(DateDiff(DateInterval.Second, startDate, enddate))

            .End()
        End With
    Else
        strResult = "No records found for specified academic year"
    End If
    Return strResult
End Function

Answer 1

您正在写入临时文件，读取该文件，并将该文件的内容写入响应。跳过临时文件并一次直接写入响应一行。这将使浏览器不会认为您的服务器超时，加快速度，并减少应用程序消耗的内存量。

之后，请查看如何在此Web请求上启用缓存，以便ASP.NET可以在多个用户在短时间内要求重新创建CSV。

Answer 2

您应该使用StringBuilder而不是连接。

Answer 3

有一些方法可以加快速度：

不要从StreamWriter写入文件到Response，直接写入页面Response。
使用datareader寻找一个解决方案而不是datatable来循环遍历数据，这些数据可能会更快。它还会降低内存使用量（不会将整个表加载到内存中）。
如果连接多个字符串使用StringBuilder，或者在这种情况下使用String.Join轻松创建整行。

String.Join示例：

For Each row in dtStudent.Rows
    Dim line as new List(of String)
    line.Add(row("fy").ToString)
    line.Add(row("uniq_stu_id").ToString)
    line.Add(-etc-)

    Response.Write(String.Join(",", line.ToArray) & vbcrlf )
Next

Answer 4

除了@Robert Levy的建议外，请注意如何使用字符串变量。在这些行上使用stringbuilder会更好：

        dim sbTemp as new StringBuilder()
        For i As Integer = 0 To dtStudent.Rows.Count - 1

            sbTemp.Append(dtStudent.Rows(i)("fy").ToString)
            sbTemp.Append(",") 
            sbTemp.Append(dtStudent.Rows(i)("uniq_stu_id").ToString)

            'etc
            sw.WriteLine(lineValue)

        Next

Answer 5

您可以研究的一件事是生产者/消费者设计模式。这可以让你做的是让一个（或多个）线程提供一个队列，该队列包含需要写入csv文件的数据和另一个执行实际写入的线程（或多个）。

更快地创建CSV文件

5 个答案: