Question

I've been following this Tutorial for impport data from a DB into LOGSTASh and create a Idex and Mapping into Elastic Search INSERT INTO LOGSTASH SELECT DATA FROM DATABASE

This is my OUTPUT based on my Configurations file:

[2017-10-12T11:50:45,807][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Users/Bruno/Downloads/logstash-5.6.2/logstash-5.6.2/modules/fb_apache/configuration"}
[2017-10-12T11:50:45,812][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Users/Bruno/Downloads/logstash-5.6.2/logstash-5.6.2/modules/netflow/configuration"}
[2017-10-12T11:50:46,518][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2017-10-12T11:50:46,521][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2017-10-12T11:50:46,652][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2017-10-12T11:50:46,654][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-10-12T11:50:46,716][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-10-12T11:50:46,734][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2017-10-12T11:50:46,749][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-10-12T11:50:47,053][INFO ][logstash.pipeline        ] Pipeline main started
[2017-10-12T11:50:47,196][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-10-12T11:50:47,817][INFO ][logstash.inputs.jdbc     ] (0.130000s) SELECT * from EP_RDA_STRING
[2017-10-12T11:50:53,095][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}

Everything seems OK, at least I think. Except the fact that querying the ES server to OUTPUT indexes and Mappings, I have it Empty.

http://localhost:9200/_all/_mapping

{}

http://localhost:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

this is my File Config:

input {
    jdbc {
        # sqlserver jdbc connection string to our database, mydb        
        jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=RDA; integratedSecurity=true;"
        # The user we wish to execute our statement as
        jdbc_user => ""
        # The path to our downloaded jdbc driver
        jdbc_driver_library => "C:\mypath\sqljdbc_6.2\enu\mssql-jdbc-6.2.1.jre8.jar"
        # The name of the driver class for Postgresql
        jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
        # our query
        statement => "SELECT * from EP_RDA_STRING"
    }
}
output {
    elasticsearch {

        index => "RDA"
        document_type => "RDA_string_view"
        document_id => "%{ndb_no}"
        hosts => "localhost:9200"
    }
}

Answer 1

您使用的是哪个版本的logstash？您用于启动logstash的命令是什么？确保输入和输出块类似于下面给出的

identifier;identifier;identifier;identifier;identifier;identifier;identifier;identifier;identifier;identifier;identifier;identifier;identifier;identifier
10;6;3;6;17;18;13;15;18;15;17;9;3;15
17;10;7;17;8;6;7;8;16;2;19;17;16;12
17;1;7;8;3;18;4;17;10;18;15;18;15;2
10;14;4;8;8;19;13;8;7;13;4;8;13;16
15;4;8;13;2;7;;1;16;7;18;17;9;2
12;;2;15;19;8;;12;18;;7;17;;5
16;;16;7;16;14;;16;19;;13;17;;16
18;;8;9;2;7;;7;6;;17;;;18
19;;1;4;5;7;;5;15;;8;;;
18;;8;10;6;19;;19;8;;9;;;
14;;16;15;;;;14;13;;;;;
9;;6;;;;;9;15;;;;;

How to create Index and Mapping into ES from LOGSTASH

1 个答案: