Question

我有3个sidekiq进程，每个进程有10个线程正在工作。问题是每个任务都执行HTTP请求（使用法拉第或HTTParty），但并发似乎被阻止了。也就是说，我在netstat -nalp中看到，一次只有3个连接到该特定端口（即使有30个工作者同时运行）。我尝试使用sidekiq的相同逻辑OUTSIDE进行Thread.ne w并且速度非常快。我觉得好像sidekiq在线程之间使用了一些池。可能吗？我尝试了HTTParty和法拉第。

我注意到，对于每个进程（不是线程），一次只有活动连接！这是问题，但是如何解决这个问题呢？

tcp        0      0 172.31.55.12:60139      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60044      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60000      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60055      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60149      xxxxxxxxxxxxxx:22225    ESTABLISHED 9702/sidekiq 4.2.10
tcp        1      0 172.31.55.12:60145      xxxxxxxxxxxxxx:22225    CLOSE_WAIT  27895/sidekiq 4.2.1
tcp        0      0 172.31.55.12:60065      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60021      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60033      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60148      xxxxxxxxxxxxxx:22225    ESTABLISHED 10234/sidekiq 4.2.1
tcp        0      0 172.31.55.12:59976      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60099      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60066      xxxxxxxxxxxxxx:22225    TIME_WAIT   -
tcp        0      0 172.31.55.12:60039      xxxxxxxxxxxxxx:22225    TIME_WAIT   -

我的代码：

def check_heartbeat(asset)
  res = Proxy.new("https://www.example.com/#{asset.external_code}/", 30).head

  archived = true if res.status == 404 && !asset.external_code.nil?
  ...

end

def perform(asset_id, asset_type)
  check_heartbeat(asset)
end

class Proxy

    NUM_OF_RETRIES = 3
    TIMEOUT = 10

    def initialize(url, timeout = TIMEOUT, retries = NUM_OF_RETRIES)
      @url = url
      @timeout = timeout
      @retries = retries
      @session_id = rand(100000).to_s
    end

    def head
      retries = 0
      begin
        conn = Faraday.new(@url, ssl: {verify: false}) do |conn|
          conn.request  :url_encoded
          conn.response :logger
          conn.use FaradayMiddleware::FollowRedirects
          conn.use  Faraday::Adapter::NetHttp
          conn.proxy("http://#{ENV['PROXY_USER'].present? ? ENV['PROXY_USER'].gsub('xxx', @session_id) : nil}:#{ENV['PROXY_PASSWORD']}@#{ENV['PROXY_HOST']}:#{ENV['PROXY_PORT']}")
        end

        conn.head

      rescue Exception => ex
        if retries < @retries          
          retries += 1
          retry
        end

        raise Exception.new("Proxy failed")
      end
    end
end

谢谢！

Answer 1

听起来像HTTParty或Faraday限制了每个主机名的打开连接数。可能想检查一下。

Sidekiq与HTTP的并发问题

1 个答案: