如何在Ruby / Faraday中使用多个线程发出持久性HTTP请求？

Question

我正在使用https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-best-practices#resiliency-considerations和faraday适配器来发出HTTP请求。

我希望通过异步执行来优化我的请求，但由于我需要持久连接，我不断收到too many connections reset这样的错误，我认为这是因为我有多线程创建新连接。

我尝试将我的适配器更改为net-http-persistent但由于连接不是持久性的，因此执行所有请求的最终结果并不符合预期。

我的目标是通过发出此HTTP请求将项目添加到购物篮。如果没有持久连接，则不会将项目添加到购物篮中。

所以，我的问题是：

是否有可能使持久性HTTP请求重用线程之间的连接？如果是这样，如何实现？

这是我的一段代码：

创建连接：

Faraday.new do |c|
        c.use :cookie_jar, jar: cookie_jar
        c.options.open_timeout = 5
        c.options.timeout = 10
        c.request :url_encoded
        c.response :logger, logger
        c.adapter :net_http_persistent do |http| # yields Net::HTTP::Persistent
          http.idle_timeout = 2
        end
      end

创建线程并获取每个线程的结果

  result = []
  threads = []
  total_items = items.size

  items.each_slice(5) do |sliced_items|

    # Create a thread for a batch of 5 items and store its result
    threads << Thread.new do
      Thread.current[:output] = browser.add_all_items(sliced_items)
    end
  end

  # Wait for all threads to finish their work and store their output into result
  threads.each do |t|
     t.join
     result << t[:output]
  end

add_all_items和add_to_cart方法：

 # Add a batch of items by the key passed (id, gtin, url)
    def add_all_items(items_info, key)
      results = []
      items_info.each do |item|
        begin
          add_to_cart(item[key], item[:quantity])
          item[:message] = nil
        rescue => e
          item[:message] = e.message
          puts "---------- BACKTRACE -------------- \n #{e.backtrace}"
        end
        puts "\n--------- MESSAGE = #{item[:message]} --------- \n"
        results << item
        puts "-------- RESULTS  #{results}"
      end
      results
    end

def add_to_cart(url, count = 1)
      response = connection.get(url) do |req|
        req.headers["User-Agent"] =  @user_agent
      end
      doc = Nokogiri::HTML(response.body)
      stoken = doc.search('form.js-oxProductForm input[name=stoken]').attr('value').value
      empty_json = '""'
      product_id = get_item_id(url)
      data = { #removed payload for security reasons }

      # Using example.com for question purposes
      response = connection.post('https://www.example.com/index.php?') do |req|
        req.headers["Origin"] = "https://www.example.com"
        req.headers["Content-Type"] = "application/x-www-form-urlencoded; charset=UTF-8"
        req.headers["Accept"] = "application/json, text/javascript, */*; q=0.01"
        req.headers["Referer"] = url
        req.headers["Pragma"] = "no-cache"
        req.headers["Accept-Language"] = "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
        req.headers["User-Agent"] = @user_agent
        req.headers["Cache-Control"] = "no-cache"
        req.headers["Connection"] = "keep-alive"
        req.headers["DNT"] ="1"
        req.headers["Content-Length"] = data.size.to_s
        req.headers["Accept"] = "*/*"
        req.headers["X-Requested-With"] = "XMLHttpRequest"
        req.headers["Connection"] = "keep-alive"
        req.body = data
      end

      begin
        json = JSON.parse(response.body)
        raise "Could not add item: #{json['message']}" if json['success'] != 1 || json['item'] != product_id
      rescue JSON::ParserError => e
        puts "JSON Error"
      end

    end

def get_item_id(url)
      response = connection.get(url) do |req|
        req.headers["User-Agent"] =  @user_agent
      end
      doc = Nokogiri::HTML(response.body)
      doc.search('.js-oxProductForm input[name=aid]').attr('value').value
end

提前致谢。