FiberError:'纤维跨线程调用'使用em-synchrony

时间:2016-01-14 18:59:08

标签: ruby multithreading eventmachine fibers em-synchrony

我在sidekiq worker中使用em-synchrony和em-http-request(Ruby 2.1)来同时抓取多个链接。一切都运作良好,除非我尝试同时使用刮刀运行多个工人。这是我使用的代码:

def process(web_pages)
  EM.synchrony do
    EM::Synchrony::Iterator.new(web_pages, concurrency=50).map do |web_page, iter|
      EventMachine::HttpRequest.new(web_page.url, request_options).get(redirects:10)
    end
    EventMachine.stop
  end
end

当同时运行多个实例运行此工作进程时,它最终会抛出以下错误:

WARN: FiberError: fiber called across threads
WARN: ruby-2.1.0/gems/em-synchrony-1.0.4/lib/em-synchrony/iterator.rb:28:in `resume'
/ruby-2.1.0/gems/em-synchrony-1.0.4/lib/em-synchrony/iterator.rb:28:in `block in map'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:172:in `call'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:172:in `block in map'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:206:in `call'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:206:in `block in inject'
/ruby-2.1.0/gems/em-synchrony-1.0.4/lib/em-synchrony/iterator.rb:17:in `call'
/ruby-2.1.0/gems/em-synchrony-1.0.4/lib/em-synchrony/iterator.rb:17:in `block in each'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:95:in `call'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:95:in `block in each'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:117:in `call'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:117:in `block (2 levels) in each'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:195:in `call'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:195:in `block (2 levels) in inject'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:161:in `call'
/ruby-2.1.0/gems/eventmachine-1.0.8/lib/em/iterator.rb:161:in `block (2 levels) in map'

0 个答案:

没有答案