如何使用Logstash将基于布尔的字段从MongoDB迁移到Elasticsearch

时间：2019-01-15 16:13:50

标签： mongodb elasticsearch logstash

我已经阅读了许多论坛以及Elasticsearch文档，但是到目前为止，我仍无法弄清楚如何使用Logstash将基于布尔的字段从MongoDB转换为Elasticsearch。

这是我的示例文档出现在MongoDB中

{
    "_id" : ObjectId("59a41f63dba6488b33268377"),
    "EntityID": "abc123"
    "IsRestrictedView" : false
},
{
    "_id" : ObjectId("59a41f63dba6488b33268388"),
    "EntityID" : "abc456",
    "IsRestrictedView" : true
},
{
    "_id" : ObjectId("59a41f63dba6488b33268399"),
    "EntityID" : "abc789",
    "IsRestrictedView" : false
 }

这是Logstash的.conf文件的内容：

input {
    mongodb {
        uri => 'mongodb://localhost/TestDB'
        placeholder_db_dir => 'E:/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 'test'
        batch_size => 5000
        generateId => false
    }
}
filter {
    mutate {
        rename => { "_id" => "mongo_id" }
    }
    mutate {
        convert => {"IsRestrictedView" => "boolean"}
    }
}
output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "mediasummary"
        doc_as_upsert => true
        document_id => "%{mongo_id}"
    }
}

Elasticsearch中的索引数据包含除“ IsRestrictedView”之外的所有其他字段。

出了什么问题？请指导。

更新：这是标准输出的一部分。

[2019-01-16T16:23:43,298][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
      "mongo_id" => "59a41f63dba6488b33268399",
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268399'), \"EntityID\"=>\"abc789\", \"IsRestrictedView\"=>false}",
          "host" => "faisal-LP",
       "logdate" => "2017-08-28T13:49:23+00:00",
      "EntityID" => "abc789",
      "@version" => "1",
    "@timestamp" => 2019-01-16T11:23:42.608Z
}
{
      "mongo_id" => "59a41f63dba6488b33268388",
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268388'), \"EntityID\"=>\"abc456\", \"IsRestrictedView\"=>true}",
          "host" => "faisal-LP",
       "logdate" => "2017-08-28T13:49:23+00:00",
      "EntityID" => "abc456",
      "@version" => "1",
    "@timestamp" => 2019-01-16T11:23:42.522Z
}
D, [2019-01-16T16:23:43.571000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268399')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:43.627000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.044s
D, [2019-01-16T16:23:43.792000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:43.802000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.009s
D, [2019-01-16T16:23:43.895000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268399')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:43.902000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.007s
D, [2019-01-16T16:23:44.035000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:44.038000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.002s
D, [2019-01-16T16:23:44.138000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268399')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:44.142000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.003s
D, [2019-01-16T16:23:44.310000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:44.315000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.003s
D, [2019-01-16T16:23:44.501000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268399')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:44.504000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.003s
D, [2019-01-16T16:23:44.588000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:44.593000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.005s
D, [2019-01-16T16:23:44.930000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268399')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:44.933000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.002s
D, [2019-01-16T16:23:45.001000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:45.007000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.004s
D, [2019-01-16T16:23:45.684000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268399')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:45.688000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.002s
D, [2019-01-16T16:23:45.866000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:45.871000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.004s
D, [2019-01-16T16:23:47.173000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268399')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:47.178000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.004s
[2019-01-16T16:23:47,291][WARN ][logstash.runner          ] SIGINT received. Shutting down.
D, [2019-01-16T16:23:47.353000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2012 type=uuid data=0x0109d90c54a44c59...>}}
D, [2019-01-16T16:23:47.358000 #15516] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.004s
[2019-01-16T16:23:49,663][FATAL][logstash.runner          ] SIGINT received. Terminating immediately..

0 个答案:

没有答案
相关问题
最新问题