异步任务 - 读取数据库数据，得到：'System.Data.SqlClient.SqlException'查询处理器无法启动必要的线程资源

Question

我在循环中创建异步任务，任务必须从sql数据库中读取数据。 随着源数据量变大，我收到了这个错误：

• 查询处理器无法为并行查询执行启动必要的线程资源。

这是我的代码：

For Each objLocDataTable In pLocationData

    If objLocDataTable IsNot Nothing AndAlso objLocDataTable.ISVALID Then

       listLocationELTTasks.Add(TaskEx.Run(
             Async Function() As Task
                    Dim resTable As System.Data.DataTable = Await mDBUtil.QueryAsync(strQry)

                    ....... 'do calculation works based on the query result
             End Function))
    END if
Next


 Public Async Function QueryAsync(ByVal strSqlQuery As String) As Task(Of DataTable)

  Dim returnDataTable As DataTable = Nothing

  Dim nTriedNumber As Integer = 0, nMaxTryNumber = 10
  Dim bStop As Boolean = False

  While (Not bStop)
   Try
    Using dbConnection As New SqlConnection(mStrConn)
     Using dbComm As New SqlCommand(strSqlQuery, dbConnection)

      Await dbConnection.OpenAsync()
      dbComm.CommandType = CommandType.Text

      Dim readerTask As Task(Of SqlDataReader) = Task.Factory.FromAsync(AddressOf dbComm.BeginExecuteReader, AddressOf dbComm.EndExecuteReader, Nothing)

      returnDataTable = Await readerTask.ContinueWith(
       Function(pervTask)
        Dim resultTable As New DataTable()
        Dim reader As SqlDataReader = Nothing
        Try
         reader = pervTask.Result
         reader.Read()
         resultTable.Load(reader)
         bStop = True
        Catch ex As Exception
         System.Diagnostics.Debug.WriteLine(String.Format("Thread ID [{0}]: nTriedNumber [{1}], QueryAsync error: {2}", Threading.Thread.CurrentThread.ManagedThreadId, nTriedNumber, ex.Message))
        Finally
         If (reader IsNot Nothing) Then
          reader.Close()
         End If
         If (Not dbConnection.State = ConnectionState.Closed) Then
          dbConnection.Close()
         End If
        End Try
        Return resultTable
       End Function)
     End Using
    End Using

   Catch ex As Exception
    System.Diagnostics.Debug.WriteLine(String.Format("Not datareader error, nTriedNumber [{0}], QueryAsync error: {1}", nTriedNumber, ex.Message))
   Finally
    'could be shorten as if( returnDataTable Is Nothing or returnDataTable.Rows.Count() = 5) Then
     System.Diagnostics.Debug.WriteLine(String.Format("Thread ID [{0}]: QueryAsync is sleeping for 1 sec", Threading.Thread.CurrentThread.ManagedThreadId))
     System.Threading.Thread.Sleep(1000)
    End If

    If (nTriedNumber >= nMaxTryNumber) Then
     System.Diagnostics.Debug.WriteLine(String.Format("Thread ID [{0}]: Reach to max try number, failed to retrieve data", Threading.Thread.CurrentThread.ManagedThreadId))
     bStop = True
    End If
   End Try
  End While

  Return returnDataTable
 End Function

和函数QueryAsync一样，我试过了 - 尽快关闭数据库连接。 - 甚至睡了一会儿 他们都没有工作

注意： - sql数据源大约是5000万条记录，我不能在for循环之前立即将它加载到内存中来进行内存中的linq查询。 我试图用Parrel.foreach替换任务，结果相同。 什么是在任务中从sql数据库读取数据的有效方法。

Answer 1

看起来您不需要异步IO。切换到同步数据库调用，您的生活变得非常简单：

pLocationData
.AsParallel()
.WithDegreeOfParallelism(8) //choose the number by testing
.ForEach(objLocDataTable => Query(objLocDataTable));

（C＃语法）。我已将QueryAsync重命名为Query，因为它应该是一个很好的旧同步方法，就像我们一直使用它一样。

这将同时执行8个线程的查询。固定数量的线程不会使DB过载。