Question

我正在制作一个必须尽快处理大约5000个字符串的程序。大约2000个这些字符串必须通过webrequest转换为mymemory.translated.net。（见下面的代码，JSON部分已删除，因为这里不需要）

Try

          url = "http://api.mymemory.translated.net/get?q=" & Firstpart & "!&langpair=de|it&de=somemail@christmas.com"

          request = DirectCast(WebRequest.Create(url), HttpWebRequest)
          response = DirectCast(request.GetResponse(), HttpWebResponse)
          myreader = New StreamReader(response.GetResponseStream())

          Dim rawresp As String
          rawresp = myreader.ReadToEnd()
          Debug.WriteLine("Raw:" & rawresp)


          Catch ex As Exception
              MessageBox.Show(ex.ToString)

          End Try

代码本身工作正常，问题是它是一个阻塞代码，每个字符串需要大约1秒。我的所有琴弦都超过了半个小时。我需要将此代码转换为非阻塞代码，并在同一时间进行多次调用。有人可以告诉我怎么做到这一点？我在考虑一个后台工作者，但这不会加快速度..它只会在不同的线程上执行代码......

谢谢！

Answer 1

如果要发送10个并行请求，则必须创建10个BackgroundWorkers。或者手动创建10个线程。然后迭代，每当完成一个worker / thread时，给它一个新的任务。

我不建议发射5000个并行线程/工作者，你必须小心：像这样的负载可能被解释为垃圾邮件或服务器的攻击。不要过度，也许与被翻译的网站交谈并向他们询问他们接受的工作量。还要考虑一下您的机器和您的上游网络可以处理的内容。

Answer 2

问题是您不仅仅受到最大并发操作数的阻碍。 HttpWebRequests本质上受到限制（我相信默认策略在任何给定时间只允许2），因此您也必须覆盖该行为。请参考下面的代码。

Imports System.Diagnostics
Imports System.IO
Imports System.Net
Imports System.Threading
Imports System.Threading.Tasks

Public Class Form1

  ''' <summary>
  ''' Test entry point.
  ''' </summary>
  Private Sub Form1_Load() Handles MyBase.Load
    ' Generate enough words for us to test thoroughput.
    Dim words = Enumerable.Range(1, 100) _
      .Select(Function(i) "Word" + i.ToString()) _
      .ToArray()

    ' Maximum theoretical number of concurrent requests.
    Dim maxDegreeOfParallelism = 24
    Dim sw = Stopwatch.StartNew()

    ' Capture information regarding current SynchronizationContext
    ' so that we can perform thread marshalling later on.
    Dim uiScheduler = TaskScheduler.FromCurrentSynchronizationContext()
    Dim uiFactory = New TaskFactory(uiScheduler)

    Dim transformTask = Task.Factory.StartNew(
      Sub()
        ' Apply the transformation in parallel.
        ' Parallel.ForEach implements clever load
        ' balancing, so, since each request won't
        ' be doing much CPU work, it will spawn
        ' many parallel streams - likely more than
        ' the number of CPUs available.
        Parallel.ForEach(words, New ParallelOptions With {.MaxDegreeOfParallelism = maxDegreeOfParallelism},
          Sub(word)
            ' We are running on a thread pool thread now.
            ' Be careful not to access any UI until we hit
            ' uiFactory.StartNew(...)

            ' Perform transformation.
            Dim url = "http://api.mymemory.translated.net/get?q=" & word & "!&langpair=de|it&de=somemail@christmas.com"
            Dim request = DirectCast(WebRequest.Create(url), HttpWebRequest)

            ' Note that unless you specify this explicitly,
            ' the framework will use the default and you
            ' will be limited to 2 parallel requests
            ' regardless of how many threads you spawn.
            request.ServicePoint.ConnectionLimit = maxDegreeOfParallelism

            Using response = DirectCast(request.GetResponse(), HttpWebResponse)
              Using myreader As New StreamReader(response.GetResponseStream())
                Dim rawresp = myreader.ReadToEnd()

                Debug.WriteLine("Raw:" & rawresp)

                ' Transform the raw response here.
                Dim processed = rawresp

                uiFactory.StartNew(
                  Sub()
                    ' This is running on the UI thread,
                    ' so we can access the controls,
                    ' i.e. add the processed result
                    ' to the data grid.
                    Me.Text = processed
                  End Sub, TaskCreationOptions.PreferFairness)
              End Using
            End Using
          End Sub)
      End Sub)

    transformTask.ContinueWith(
      Sub(t As Task)
        ' Always stop the stopwatch.
        sw.Stop()

        ' Again, we are back on the UI thread, so we
        ' could access UI controls if we needed to.
        If t.Status = TaskStatus.Faulted Then
          Debug.Print("The transformation errored: {0}", t.Exception)
        Else
          Debug.Print("Operation completed in {0} s.", sw.ElapsedMilliseconds / 1000)
        End If
      End Sub,
      uiScheduler)
  End Sub

End Class

Answer 3

我会为每个请求创建一个任务，因此您可以使用ContinueWith为每个呼叫设置回叫：

  For Each InputString As String In myCollectionString


            Tasks.Task(Of String).Factory.StartNew(Function(inputString)

                    Dim request As HttpWebRequest
                    Dim myreader As StreamReader
                    Dim response As HttpWebResponse
                    Dim rawResp As String = String.Empty

                    Try

                      Dim url As String = "http://api.mymemory.translated.net/get?q=" & inputString & "!&langpair=de|it&de=somemail@christmas.com"

                      request = DirectCast(WebRequest.Create(url), HttpWebRequest)
                      response = DirectCast(request.GetResponse(), HttpWebResponse)
                      myreader = New StreamReader(response.GetResponseStream())

                      rawResp = myreader.ReadToEnd()
                      Debug.WriteLine("Raw:" & rawResp)


                    Catch ex As Exception
                      MessageBox.Show(ex.ToString)

                     End Try

                     Return rawResp

              End Function, CancellationToken.None, _ 
              Tasks.TaskCreationOptions.None).ContinueWith _
              (Sub(task As Tasks.Task(Of String))                                                                                                 
                'Dom something with result                                                                                                                          
                 Console.WriteLine(task.Result)                                                                                                                     
              End Sub)    

        Next

非阻塞webrequests vb.net

3 个答案: