Question

我们有大约四百万个zip文件，每个包含两个图像。这些zip文件作为BLOB存储在Oracle表中。要求是调整这些图像的大小，并将它们作为BLOB（但现在是JPEG）插入到不同的表中。因此，我所做的是： 1.检索每个zip文件并将其保存到文件系统。解压缩它 3.获取图像并调整其大小 4.将新JPEG插入新表。 5.删除包含图像的zip文件和uznipped文件夹。

对于1000个zip文件（本地），此任务大约需要三分钟，因此对于400万个拉链，大约需要8.3天（如果没有超时问题等）。这样做不需要大量存储，因为每个zip在处理后都会被删除。内存和处理消耗也很好。

我正在寻找的是一种优化我的功能的方法。我正在使用异步函数，但它们的构造方式不仅节省了时间，而且只节省了资源。任何想法如何使整个过程更快？

这就是我调用我的功能的方式

Await Cheque.CopyBinaryValueToFile()

我的功能：

' Application retrieving a large BLOB from Oracle using the new asynchronous capability
    Public Shared Async Function CopyBinaryValueToFile() As Task
        '{Time count
        Dim start_time As DateTime
        Dim stop_time As DateTime
        Dim elapsed_time As TimeSpan
        start_time = Now
        '}

        Dim sCurrentZip As String
        Dim sRecordID As String
        Dim sTempFolder As String = "TEMP"
        Dim oExtractedFiles As New ExtractedFiles

        'Create temporary folder for the zip files
        If Not Directory.Exists("TEMP") Then
            Directory.CreateDirectory("TEMP")
        End If

    Using connection As New OracleConnection(SettingsConnection.oracleCon)
        Await connection.OpenAsync()
        Using command As New OracleCommand("SELECT ch.RECORDID, ch.IMAGES FROM CHEQUES ch WHERE ch.RECORDID NOT IN (SELECT res.RECORDID FROM CHEQUES_RESIZED res)", connection)

            ' The reader needs to be executed with the SequentialAccess behavior to enable network streaming
            ' Otherwise ReadAsync will buffer the entire BLOB into memory which can cause scalability issues or even OutOfMemoryExceptions
            Using reader As OracleDataReader = Await command.ExecuteReaderAsync(CommandBehavior.SequentialAccess)
                While Await reader.ReadAsync()
                    If reader.HasRows Then
                        If Not (Await reader.IsDBNullAsync(1)) Then
                            sRecordID = CStr(reader("RecordID"))
                            sCurrentZip = Path.Combine(sTempFolder, sRecordID & ".zip")  'specify filename which is the TEMP\recordid + .zip
                            Using file As New FileStream(sCurrentZip, FileMode.Create, FileAccess.Write)
                                Using data As Stream = reader.GetStream(1)
                                    ' Asynchronously copy the stream from the server to the file we just created
                                    Await data.CopyToAsync(file)
                                    ' Unzip the file and store data related to the unzip to the object of ExtractedFiles
                                    oExtractedFiles = Await UnzipFile(sTempFolder, sCurrentZip, sRecordID)
                                End Using
                            End Using

                            If oExtractedFiles.IsSuccessful Then
                                'If they are successfully inserted to the resized table, then delete the leftover files
                                If Await StreamBLOBToServer(oExtractedFiles) Then
                                    Await DeleteExtractedData(oExtractedFiles)
                                End If
                            End If
                        End If
                    End If
                End While
            End Using
        End Using
    End Using
    '{Time count
    stop_time = Now
    elapsed_time = stop_time.Subtract(start_time)
    MessageBox.Show("elapsed_time = " & stop_time.Subtract(start_time).ToString & Environment.NewLine _
    & elapsed_time.TotalSeconds.ToString("0.000000"))
    '}
End Function

优化技术 - 异步功能所需的建议

0 个答案: