Question

在Heroku上我有一个rails应用程序运行，包括几个web dynos以及一个worker dyno。我在Sidekiq上全天运行成千上万的工作任务，但偶尔会引发ActiveRecord :: ConnectionTimeoutError（大约每天50次）。我按照以下方式设置了我的独角兽服务器

worker_processes 4
timeout 30
preload_app true

before_fork do |server, worker|
    # As suggested here: https://devcenter.heroku.com/articles/rails-unicorn
    Signal.trap 'TERM' do
        puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
        Process.kill 'QUIT', Process.pid
    end

    if defined?(ActiveRecord::Base)
        ActiveRecord::Base.connection.disconnect!
    end
end

after_fork do |server,worker|
    if defined?(ActiveRecord::Base)
        config = Rails.application.config.database_configuration[Rails.env]
        config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
        config['pool']            = ENV['DB_POOL'] || 10
        ActiveRecord::Base.establish_connection(config)
    end

    Sidekiq.configure_client do |config|
        config.redis = { :size => 1 }
    end

    Sidekiq.configure_server do |config|
        config = Rails.application.config.database_configuration[Rails.env]
        config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
        config['pool']            = ENV['DB_POOL'] || 10
        ActiveRecord::Base.establish_connection(config)
    end
end

在heroku上，我将DB_POOL配置变量设置为2 recommended by Heroku。这些错误应该发生吗？似乎很奇怪，这是不可能避免这样的错误，不是吗？你会建议什么？

Answer 1

sidekiq服务器（服务器上运行的进程实际执行延迟的任务）默认情况下最多可以拨打25个线程来处理其队列中的工作。如果您的任务需要，这些线程中的每一个都可以通过ActiveRecord请求与主数据库的连接。

如果你只有一个连接池有5个连接，但你有25个线程试图连接，5秒后线程将放弃，如果他们无法从池中获得可用的连接，你将得到一个连接超时错误。

将Sidekiq服务器的池大小设置为更接近并发级别（在启动过程时使用-c标志设置）将有助于缓解此问题，但代价是打开更多数据库连接。例如，如果您使用Heroku并使用Postgres，他们的一些计划限制为20，而其他计划的连接限制为500（source）。

如果您正在运行像Unicorn这样的多进程服务器环境，则还需要监视每个分叉进程所做的连接数。如果您有4个独角兽进程，并且默认连接池大小为5，那么任何给定时间的独角兽环境都可能有20个实时连接。您可以在Heroku's docs上详细了解相关信息。另请注意，数据库池大小并不意味着每个dyno现在都有那么多打开的连接，但只有在需要新连接时才会创建它，直到最多创建了多个连接。

话虽如此，这就是我的所作所为。

# config/initializers/unicorn.rb

if ENV['RACK_ENV'] == 'development'
  worker_processes 1
  listen "#{ENV['BOXEN_SOCKET_DIR']}/rails_app"
  timeout 120
else
  worker_processes Integer(ENV["WEB_CONCURRENCY"] || 2)
  timeout 29
end

# The timeout mechanism in Unicorn is an extreme solution that should be avoided whenever possible. 
# It will help catch bugs in your application where and when your application forgets to use timeouts,
# but it is expensive as it kills and respawns a worker process.
# see http://unicorn.bogomips.org/Application_Timeouts.html

# Heroku recommends a timeout of 15 seconds. With a 15 second timeout, the master process will send a 
# SIGKILL to the worker process if processing a request takes longer than 15 seconds. This will 
# generate a H13 error code and you’ll see it in your logs. Note, this will not generate any stacktraces 
# to assist in debugging. Using Rack::Timeout, we can get a stacktrace in the logs that can be used for
# future debugging, so we set that value to something less than this one

preload_app true # for new relic

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end

end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to sent QUIT'
  end

  Rails.logger.info("Done forking unicorn processes")

  #https://devcenter.heroku.com/articles/concurrency-and-database-connections
  if defined?(ActiveRecord::Base)

    db_pool_size = if ENV["DB_POOL"]
      ENV["DB_POOL"]
    else
      ENV["WEB_CONCURRENCY"] || 2
    end

    config = Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    config['pool']              = ENV['DB_POOL'] || 2
    ActiveRecord::Base.establish_connection(config)

    # Turning synchronous_commit off can be a useful alternative when performance is more important than exact certainty about the durability of a transaction
    ActiveRecord::Base.connection.execute "update pg_settings set setting='off' where name = 'synchronous_commit';"    

    Rails.logger.info("Connection pool size for unicorn is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
  end

end

对于sidekiq：

# config/initializers/sidekiq.rb

Sidekiq.configure_server do |config|

  sidekiq_pool = ENV['SIDEKIQ_DB_POOL'] || 20

  if defined?(ActiveRecord::Base)
    Rails.logger.debug("Setting custom connection pool size of #{sidekiq_pool} for Sidekiq Server")
    db_config = Rails.application.config.database_configuration[Rails.env]
    db_config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    cb_config['pool']              = sidekiq_pool
    ActiveRecord::Base.establish_connection(db_config)

    Rails.logger.info("Connection pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
  end
end

如果一切顺利，当您启动流程时，您会看到日志中的内容：

Setting custom connection pool size of 10 for Sidekiq Server
Connection pool size for Sidekiq Server is now: 20
Done forking unicorn processes
   (1.4ms)  update pg_settings set setting='off' where name = 'synchronous_commit';
Connection pool size for unicorn is now: 2

来源：

Answer 2

对于Sidekiq服务器配置，建议使用与您的并发数相同的db_pool数字，我假设您将其设置为大于2。

假设您的db_pool设置正在unicorn.rb（我没有经验这样做），可能的解决方案是设置另一个环境变量来控制Sidekiq db_pool直接

如果你的sidekiq并发度为20，那么就像：

配置var - SIDEKIQ_DB_POOL = 20

Sidekiq.configure_server do |config|
  config = Rails.application.config.database_configuration[Rails.env]
  config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
  config['pool']            = ENV['SIDEKIQ_DB_POOL'] || 10
  ActiveRecord::Base.establish_connection(config)
end

这可确保您有两个针对您的网络工作人员DB_POOL和后台工作人员SIDEKIQ_DB_POOL优化的独立池

可以避免Heroku上的ActiveRecord :: ConnectionTimeoutError？

2 个答案: