在PHP中打开和关闭16000个Mongo数据库实例

时间:2012-01-14 21:37:53

标签: php mongodb nosql

我目前正在进行一些个人测试和基准测试,以比较使用MongoDB和MySQL与实际示例数据之间的工作流程和效率。

要在每个数据库中设置我的数据,我正在做几千个循环并随机创建数据对象以插入到数据库中。

但是我在PHP中使用Mongo类时遇到了一些问题,我无法解决。问题是这样的:

我有一个循环,它创建一个新的Mongo实例和连接,将一个小数组插入一个集合,然后关闭连接。此循环应运行20000次。然而,当它试图创建实例/建立连接时,它始终在16300nd循环(最小值为16200,最大值为16350,我会说几次运行后)失败。

循环中的代码如下:

$data = get_random_user_data();

$mongo = new Mongo('mongodb://admin:password@localhost:27017/test');

    if ($mongo->test->users->insert($data)) {
        $users[] = array('id' => $data['_id'], 'name' => $data['username']);
    echo $i." - Added user: ".$data['username'].'<br/>';
    }

$mongo->close();

get_random_user_data()只返回一个简单的关联数组。

我得到的错误是:

Fatal error: Uncaught exception 'MongoConnectionException' with message 'Unknown error'

在线:

$mongo = new Mongo('mongodb://admin:password@localhost:27017/test');

有什么想法吗?是否存在一些我缺少的基本安全或垃圾邮件预防措施?

提前致谢。

额外信息:

脚本在大约114.9797秒死亡。这不是一个PHP内存或基于时间的问题,因为提出了所有限制,我运行我的MySQL基准测试昨天插入120000行(使用相同的方法循环打开连接,插入,关闭连接)大约一个小时没有问题。

运行PHP 5.3.5版

phpinfo Mongo信息:

MongoDB Support enabled
Version 1.2.0-
Directive   Local Value Master Value
mongo.allow_empty_keys  0   0
mongo.allow_persistent  1   1
mongo.auto_reconnect    1   1
mongo.chunk_size    262144  262144
mongo.cmd   $   $
mongo.default_host  localhost   localhost
mongo.default_port  27017   27017
mongo.long_as_object    0   0
mongo.native_long   0   0
mongo.no_id 0   0
mongo.utf8  1   1

3 个答案:

答案 0 :(得分:6)

您的操作系统只有有限数量的插槽才能打开。当您打开一个套接字然后关闭它时,操作系统不会立即将其放回“可用”池中,它会在“等待时间”状态下挂起一段时间,Nat提到in his answer

您可以增加操作系统打开的套接字数量,请参阅http://www.mongodb.org/display/DOCS/Too+Many+Open+Files(每个套接字都是一个打开的“文件”)。

此外,您使用的是旧版本的驱动程序,您可能需要考虑升级。

答案 1 :(得分:3)

我确认这也是ruby驱动程序的相同行为。我用ruby mongo复制了类似的情况,每次通过打开/关闭mongo实例来插入20000条记录。它在14110到14200之间继续失败。

这是错误消息

 uncaught throw #<Mongo::ConnectionFailure: Failed to connect to host localhost and port 27017: Cannot assign requested address - connect(2)>

这是完整的堆栈跟踪

Failed to connect to host localhost and port 27017: Cannot assign requested address - connect(2)
["/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:171:in `rescue in checkout_new_socket'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:166:in `checkout_new_socket'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:267:in `block (2 levels) in checkout'", "<internal:prelude>:10:in `synchronize'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:259:in `block in checkout'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:252:in `loop'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/pool.rb:252:in `checkout'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/connection.rb:496:in `checkout_writer'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/networking.rb:34:in `send_message'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/collection.rb:948:in `block in insert_documents'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/util/logging.rb:28:in `instrument'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/collection.rb:944:in `insert_documents'", "/home/ramesh/.rvm/gems/ruby-1.9.2-p290/gems/mongo-1.5.2/lib/mongo/collection.rb:343:in `insert'", "/home/ramesh/Desktop/load_test.rb:15:in `block in load_test'", "/home/ramesh/Desktop/load_test.rb:6:in `times'", "/home/ramesh/Desktop/load_test.rb:6:in `load_test'", "(irb):2:in `irb_binding'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/workspace.rb:80:in `eval'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/workspace.rb:80:in `evaluate'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/context.rb:254:in `evaluate'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:159:in `block (2 levels) in eval_input'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:273:in `signal_status'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:156:in `block in eval_input'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:243:in `block (2 levels) in each_top_level_statement'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:229:in `loop'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:229:in `block in each_top_level_statement'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:228:in `catch'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb/ruby-lex.rb:228:in `each_top_level_statement'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:155:in `eval_input'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:70:in `block in start'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:69:in `catch'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/irb.rb:69:in `start'", "/home/ramesh/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'"]
ArgumentError: uncaught throw #<Mongo::ConnectionFailure: Failed to connect to host localhost and port 27017: Cannot assign requested address - connect(2)>
    from /home/ramesh/Desktop/load_test.rb:21:in `throw'
    from /home/ramesh/Desktop/load_test.rb:21:in `rescue in block in load_test'
    from /home/ramesh/Desktop/load_test.rb:7:in `block in load_test'
    from /home/ramesh/Desktop/load_test.rb:6:in `times'
    from /home/ramesh/Desktop/load_test.rb:6:in `load_test'

同样@Chrisui说,我也没有遇到任何内存下降。

这是我试过的剧本

require 'mongo'

def load_test
    start = Time.now
    puts 'starting at :' + start.to_s 
    20000.times do |i|    
            begin
                puts i      
            doc = {"name" => "MongoDB", "type" => "database", "count" => 1,"info" => {"x" => 203, "y" => '102'}}
            con = Mongo::Connection.new("localhost")
            db   = con['bulktest']
            coll = db['test']
            coll.insert(doc)
            con.close
                con,db,coll=nil,nil,nil
        rescue Exception => e  
            puts e.message  
            puts e.backtrace.inspect  
            throw e 
        end

    end
    stop = Time.now
    puts 'stoping at :' + stop.to_s 
    puts 'elapsed time is ' + (stop-start).to_s + ' seconds'
end

答案 2 :(得分:1)

您的基准没有意义。不要保持打开/关闭连接。您应该重复使用持久连接而不是每次打开/关闭。如果你快速打开和关闭套接字,你将有太多的套接字处于定时等待状态仍然使用文件描述符