异步函数调用与油门并行

时间:2018-03-27 14:02:40

标签: parallel-processing f# queue message-queue throttling

我根据这个建议尝试但仍然有打嗝,有人可以根据尝试提供帮助吗?我的方法签名是不同的返回一个元组,我需要将结果返回给调用者(而不仅仅是执行)

我在这行do! work

中出错了
  
    

错误:表达式应该具有Result类型,但这里的类型为“unit”

  
  type Error = {
    code : int
    message : string
  }

  /// Message type used by the agent - contains queueing
  /// of work items and notification of completion
  type internal ThrottlingAgentMessage =
    | Completed of Result<string, Error>
    | Enqueue of Async<Result<string, Error>>

  /// Represents an agent that runs operations in concurrently. When the number
  /// of concurrent operations exceeds 'limit', they are queued and processed later
  let throttlingAgent limit =
    MailboxProcessor.Start(fun inbox ->
      async {
        // The agent body is not executing in parallel,
        // so we can safely use mutable queue & counter
        let queue = System.Collections.Generic.Queue<Async<Result<string, Error>>>()
        let running = ref 0

        while true do

          // Enqueue new work items or decrement the counter
          // of how many tasks are running in the background
          let! msg = inbox.Receive()
          match msg with
          | Completed r -> decr running
          | Enqueue w -> queue.Enqueue(w)

          // If we have less than limit & there is some work to
          // do, then start the work in the background!
          while running.Value < limit && queue.Count > 0 do
            let work = queue.Dequeue()
            incr running
            do! // When the work completes, send 'Completed'
                // back to the agent to free a slot
                async {
                  do! work
                  inbox.Post(Completed)
                }
                |> Async.StartChild
                |> Async.Ignore
      })

  let requestAsync (url: string) : Async<Result<string, Error>> =
      async {
          Console.WriteLine ("Simulating request " + url)
          try
              do! Async.Sleep(1000)
              return Ok (url + ":body...")
          with :? WebException as e ->
              return Error {code = 500; message = "Internal Server Error";}
      }

  let urls = [|
              "http://www.example.com/1";
              "http://www.example.com/2";
              "http://www.example.com/3";
              "http://www.example.com/4";
              "http://www.example.com/5";
              "http://www.example.com/6";
              "http://www.example.com/7";
              "http://www.example.com/8";
              "http://www.example.com/9";
          |]

  let w = throttlingAgent 3
  for url in urls do
    requestAsync url
    |> Enqueue
    |> w.Post

1 个答案:

答案 0 :(得分:2)

我认为问题在于您尝试修改代理以使工作项不是Async<unit>,即在不返回结果的情况下运行和完成的操作,而是实际返回某些内容的操作。你可以这样做,但你必须决定你想对结果做些什么。

在这种情况下,我认为按原样保留代理更容易,而是在最后处理结果集合。例如,如果要在集合中收集它们,可以写下:

let results = System.Collections.Concurrent.ConcurrentBag<_>()
let w = throttlingAgent 3
for url in urls do
  async { 
    let! res = requestAsync url
    results.Add res } |> Enqueue |> w.Post

为了完整性,这是使用以下类型和代理的定义:

type Error = {
  code : int
  message : string
}

/// Message type used by the agent - contains queueing
/// of work items and notification of completion
type ThrottlingAgentMessage =
  | Completed 
  | Enqueue of Async<unit>

/// Represents an agent that runs operations in concurrently. When the number
/// of concurrent operations exceeds 'limit', they are queued and processed later
let throttlingAgent limit =
  MailboxProcessor.Start(fun inbox ->
    async {
      // The agent body is not executing in parallel,
      // so we can safely use mutable queue & counter
      let queue = System.Collections.Generic.Queue<Async<unit>>()
      let running = ref 0

      while true do

        // Enqueue new work items or decrement the counter
        // of how many tasks are running in the background
        let! msg = inbox.Receive()
        match msg with
        | Completed -> decr running
        | Enqueue w -> queue.Enqueue(w)

        // If we have less than limit & there is some work to
        // do, then start the work in the background!
        while running.Value < limit && queue.Count > 0 do
          let work = queue.Dequeue()
          incr running
          do! // When the work completes, send 'Completed'
              // back to the agent to free a slot
              async {
                do! work
                inbox.Post(Completed)
              }
              |> Async.StartChild
              |> Async.Ignore
    })

let requestAsync (url: string) : Async<Result<string, Error>> =
    async {
        Console.WriteLine ("Simulating request " + url)
        try
            do! Async.Sleep(1000)
            return Ok (url + ":body...")
        with :? WebException as e ->
            return Error {code = 500; message = "Internal Server Error";}
    }