ActiveRecord :: ConnectionTimeoutError(无法在5.000秒内获取数据库连接(等待5.000秒))

时间:2015-12-29 16:04:39

标签: mysql ruby-on-rails ruby-on-rails-4 activerecord

我在Rails 4应用程序中收到此严重错误,因此我与数据库的连接丢失。

F, [2015-12-23T18:06:22.875935 #13919] FATAL -- :
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5.000 seconds (waited 5.009 seconds)):
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `block in wait_poll'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `loop'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `wait_poll'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:136:in `block in poll'
  /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:146:in `synchronize'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:134:in `poll'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:423:in `acquire_connection'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:356:in `block in checkout'
  /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:355:in `checkout'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
  /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:546:in `retrieve_connection'
  activerecord (4.0.0) lib/active_record/connection_handling.rb:79:in `retrieve_connection'
  activerecord (4.0.0) lib/active_record/connection_handling.rb:53:in `connection'
  activerecord (4.0.0) lib/active_record/query_cache.rb:51:in `restore_query_cache_settings'
  activerecord (4.0.0) lib/active_record/query_cache.rb:43:in `rescue in call'
  activerecord (4.0.0) lib/active_record/query_cache.rb:32:in `call'
  activerecord (4.0.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:626:in `call'
  actionpack (4.0.0) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
  activesupport (4.0.0) lib/active_support/callbacks.rb:373:in `_run__285615481658568074__call__callbacks'
  activesupport (4.0.0) lib/active_support/callbacks.rb:80:in `run_callbacks'
  actionpack (4.0.0) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  actionpack (4.0.0) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
  actionpack (4.0.0) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
  actionpack (4.0.0) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
railties (4.0.0) lib/rails/rack/logger.rb:38:in `call_app'
  railties (4.0.0) lib/rails/rack/logger.rb:21:in `block in call'
  activesupport (4.0.0) lib/active_support/tagged_logging.rb:67:in `block in tagged'
  activesupport (4.0.0) lib/active_support/tagged_logging.rb:25:in `tagged'
  activesupport (4.0.0) lib/active_support/tagged_logging.rb:67:in `tagged'
  railties (4.0.0) lib/rails/rack/logger.rb:21:in `call'
  actionpack (4.0.0) lib/action_dispatch/middleware/request_id.rb:21:in `call'
  rack (1.5.2) lib/rack/methodoverride.rb:21:in `call'
  rack (1.5.2) lib/rack/runtime.rb:17:in `call'
  activesupport (4.0.0) lib/active_support/cache/strategy/local_cache.rb:83:in `call'
  railties (4.0.0) lib/rails/engine.rb:511:in `call'
  railties (4.0.0) lib/rails/application.rb:97:in `call'
  puma (2.10.1) lib/puma/configuration.rb:74:in `call'
  puma (2.10.1) lib/puma/server.rb:490:in `handle_request'
  puma (2.10.1) lib/puma/server.rb:361:in `process_client'
  puma (2.10.1) lib/puma/server.rb:254:in `block in run'
  puma (2.10.1) lib/puma/thread_pool.rb:96:in `call'
  puma (2.10.1) lib/puma/thread_pool.rb:96:in `block in spawn_thread'

我正在尝试调试它,但它非常复杂。我不知道问题的来源。

我在ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)之前阅读了这篇文章,并且@railsana的答案符合我的一个可能情景。

我有一些批处理过程,它通过计划任务从模型(外部控制器)调用大量查询。我试图按照他们的建议并将此代码添加到此功能中,但问题仍然存在。

ActiveRecord::Base.connection_pool.with_connection do
  # your code
end

以防万一是相关信息,我的rails应用程序在Puma服务器上运行,我的数据库是MySQL。

任何建议,它可以是什么,或如何调试它?该错误实际上是致命的,因为我的应用程序无法处理任何查询。

更新 回答问题: 1.在生产中,我在config / database.yml中配置了一个带有5个连接的池。 2.我的Puma服务器启动日志:

Puma starting in single mode...
* Version 2.10.1 (ruby 2.1.2-p95), codename: Robots on Comets
* Min threads: 0, max threads: 16
* Environment: production
* Daemonizing..

只启动了一名工人。

更新2:

检查puma.stderr.log我发现了这种错误日志:

{ 70238458077400 rufus-scheduler intercepted an error:
  70238458077400   job:
  70238458077400     Rufus::Scheduler::EveryJob "90s" {}
  70238458077400   error:
  70238458077400     70238458077400
  70238458077400     Net::ReadTimeout
  70238458077400     Net::ReadTimeout
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/protocol.rb:158:in `rescue in rbuf_fill'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/protocol.rb:152:in `rbuf_fill'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/protocol.rb:134:in `readuntil'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/protocol.rb:144:in `readline'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http/response.rb:39:in `read_status_line'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http/response.rb:28:in `read_new'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:1408:in `block in transport_request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:1405:in `catch'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:1405:in `transport_request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:1378:in `request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb:65:in `perform'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb:42:in `block in request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb:78:in `call'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb:78:in `block in do_request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:853:in `start'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb:76:in `do_request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb:33:in `request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi.rb:161:in `request'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi.rb:133:in `post'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/operation.rb:94:in `block in call_with_logging'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/request_logger.rb:12:in `call'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/request_logger.rb:12:in `log'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/operation.rb:94:in `call_with_logging'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/operation.rb:54:in `call'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/client.rb:36:in `call'
  70238458077400       /home/ubuntu/env/production/www/yanpyapi/lib/mmk_service.rb:560:in `block (2 levels) in getAvailabilities'
  70238458077400       /home/ubuntu/env/production/www/yanpyapi/lib/mmk_service.rb:544:in `each'
  70238458077400       /home/ubuntu/env/production/www/yanpyapi/lib/mmk_service.rb:544:in `block in getAvailabilities'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-4.0.0/lib/active_record/relation/delegation.rb:13:in `each'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-4.0.0/lib/active_record/relation/delegation.rb:13:in `each'
  70238458077400       /home/ubuntu/env/production/www/yanpyapi/lib/mmk_service.rb:530:in `getAvailabilities'
  70238458077400       /home/ubuntu/env/production/www/yanpyapi/lib/mmk_service.rb:65:in `updateAvailabilities'
  70238458077400       /home/ubuntu/env/production/www/yanpyapi/app/models/secretary.rb:182:in `block in getBoatsAvailability'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection'
  70238458077400       /home/ubuntu/env/production/www/yanpyapi/app/models/secretary.rb:181:in `getBoatsAvailability'
  70238458077400       /home/ubuntu/env/production/www/yanpyapi/config/initializers/task_scheduler.rb:29:in `block in <top (required)>'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rufus-scheduler-3.0.3/lib/rufus/scheduler/jobs.rb:224:in `call'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rufus-scheduler-3.0.3/lib/rufus/scheduler/jobs.rb:224:in `do_trigger'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rufus-scheduler-3.0.3/lib/rufus/scheduler/jobs.rb:269:in `block (3 levels) in start_work_thread'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rufus-scheduler-3.0.3/lib/rufus/scheduler/jobs.rb:272:in `call'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rufus-scheduler-3.0.3/lib/rufus/scheduler/jobs.rb:272:in `block (2 levels) in start_work_thread'
 70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rufus-scheduler-3.0.3/lib/rufus/scheduler/jobs.rb:258:in `loop'
  70238458077400       /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rufus-scheduler-3.0.3/lib/rufus/scheduler/jobs.rb:258:in `block in start_work_thread'
} 70238458077400 .

这是发送抛出此异常的代码部分(我认为没有捕获异常):

 # This method is in a model invoked by an scheduled task rufus-scheduler that is executed every 90 seconds.

 def self.getBoatsAvailability
    if Rails.env.production?      
      require 'service'
      # This line was added to try to fix this problem (I read this approach in another post link above)
      ActiveRecord::Base.connection_pool.with_connection do
        Service.getAvailabilities(false)
      end
    end
 end

def self.getAvailabilities(isMonthly)
    @logger.debug "Importing availabilities..." 

    # Savon is a service to communicate with SOAP web services    
    client = Savon.client(wsdl: "url", 
                          log_level: :debug,
                          log: true,
                          pretty_print_xml: true)
    mmkCompanies = Mmk::Company.select("id").where(loading: true).order("id") 
    createdAt = Time.now        
    @logger.debug "getAvailabilities Before each company."
    for mmkCompany in mmkCompanies do
      lastModifiedReq = Mmk::Availability.find_by_sql(["SELECT max(a.created_at) as max_date
                                                        FROM mmk_availabilities a, mmk_resources r 
                                                        WHERE a.resource_id = r.id
                                                        AND r.company_id = ?", mmkCompany.id]).first.max_date                
      (0..1).each do |i|
        year = Time.now.year + i        
        message = {'in0' => credentials, 
                   'in1' => credentials, 
                   'in2' => credentials,
                   'in3' => mmkCompany.id,
                   'in4' => year,
                   'in5' => true,              
                   'in6' => lastModifiedReq}

        @logger.debug "getAvailabilities Before client call."   
        response = client.call(:get_availability_info, message: message)
        @logger.debug "getAvailabilities After client call."    
        availabilitiesXML = response.to_hash[:get_availability_info_response][:out]
        availabilitiesParsed = Nokogiri::XML(availabilitiesXML)
        availabilities = availabilitiesParsed.xpath("//reservation") 
        @logger.debug "getAvailabilities Before transaction."             
        Mmk::Availability.transaction do
          @logger.debug "getAvailabilities Transaction opened."   
          for availability in availabilities do 
            id = availability["id"]           
            # More parse parameters unrelevant code              
            mmkAvailability = Mmk::Availability.find_or_initialize_by(id: id)                   
            mmkAvailability.update(resource_id: resourceId, status: status, blocks_availability: blocksAvailability,
                                   date_from: dateFrom, date_to: dateTo, base_from: baseFrom, base_to: baseTo, 
                                   option_expiry_date: optionExpiryDate, last_modified: lastModified, created_at: createdAt)
            mmkAvailability.save            
          end
        end
        @logger.debug "getAvailabilities After transaction (closed)?."
      end
    end
    @logger.debug "Availability imported successfully."
  end

此行例外:

`70238458077400       /home/ubuntu/env/production/www/yanpyapi/lib/mmk_service.rb:560:in `block (2 levels) in getAvailabilities'`

对应于:

response = client.call(:get_availability_info, message: message)

我的分析(并非一切都与我匹配)是: - 方法getAvailabilities正在向第三方系统发送Web服务SOAP请求。 - 有时由于某种原因,此连接丢失或第三方服务没有响应并且抛出Net :: ReadTimeout异常。 - 未捕获此异常。并且数据库连接(这是我不理解的部分)保持打开状态。 - 当这个问题发生5次时,池连接数为0,我遇到了主要问题。

1 个答案:

答案 0 :(得分:0)

要调试此问题,我首先要在rescue方法中添加异常处理程序(self.getAvailabilities子句)。

def self.getAvailabilities(isMonthly)
  ... # existing code goes here

rescue Net::ReadTimeout => err
  # I would use pry to put a breakpoint here and see what is keeping the DB connection open.
  binding.pry
end

使用pry gem在处理程序中插入断点。一旦Savon呼叫超时,它应该击中该断点。然后,您可以使用REPL来确定谁持有连接,并可能关闭它。见这里:How to find current connection pool size on heroku

如果未捕获到异常,我会将rescue行更改为捕获StandardError而不是Net::ReadTimeout,然后重试。