Question

我想将csv加载到elasticSearch中。我已按照此博客中详述的步骤进行操作：http://blog.webkid.io/visualize-datasets-with-elk/

我使用的是elasticsearch 2.3.1，kibana-4.5.0-linux-x64，logstash-2.3.1。我已经删除了sincedb文件，它位于我的/ home / ec2-user /目录中。以下是logstash-csv.conf文件：

input {
  file {
    path => "/etc/logstash/data.csv"
    type => "core2"
    start_position => "beginning"
  }
}
filter {
  csv {
      separator => ","
      columns => ["Date","Open","High","Low","Close","Volume","Adj Close"]
  }
  mutate {convert => ["High", "float"]}
  mutate {convert => ["Open", "float"]}
  mutate {convert => ["Low", "float"]}
  mutate {convert => ["Close", "float"]}
  mutate {convert => ["Volume", "float"]}
}
output {  
    elasticsearch {
        action => "index"
        hosts => "localhost:9200"
        index => "stock"
    workers => 1
    }
    stdout {}
}

以下是我在命令行提供curl 'localhost:9200/_cat/indices?v'时得到的内容：

health status index   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibi     1   1          4            0     20.1kb         20.1kb
yellow open   .kibana   1   1          2            0      8.3kb          8.3kb

为什么没有创建索引？还在输出下方提供 - verbose 选项：

starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 2
Registering file input {:path=>["/etc/logstash/data.csv"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/home/ec2-user/.sincedb_07be1b1b016d7137746afa513c111dad", :path=>["/etc/logstash/data.csv"], :level=>:info}
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true, "ignore_above"=>256}}}}}, {"float_fields"=>{"match"=>"*", "match_mapping_type"=>"float", "mapping"=>{"type"=>"float", "doc_values"=>true}}}, {"double_fields"=>{"match"=>"*", "match_mapping_type"=>"double", "mapping"=>{"type"=>"double", "doc_values"=>true}}}, {"byte_fields"=>{"match"=>"*", "match_mapping_type"=>"byte", "mapping"=>{"type"=>"byte", "doc_values"=>true}}}, {"short_fields"=>{"match"=>"*", "match_mapping_type"=>"short", "mapping"=>{"type"=>"short", "doc_values"=>true}}}, {"integer_fields"=>{"match"=>"*", "match_mapping_type"=>"integer", "mapping"=>{"type"=>"integer", "doc_values"=>true}}}, {"long_fields"=>{"match"=>"*", "match_mapping_type"=>"long", "mapping"=>{"type"=>"long", "doc_values"=>true}}}, {"date_fields"=>{"match"=>"*", "match_mapping_type"=>"date", "mapping"=>{"type"=>"date", "doc_values"=>true}}}, {"geo_point_fields"=>{"match"=>"*", "match_mapping_type"=>"geo_point", "mapping"=>{"type"=>"geo_point", "doc_values"=>true}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "doc_values"=>true}, "@version"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip", "doc_values"=>true}, "location"=>{"type"=>"geo_point", "doc_values"=>true}, "latitude"=>{"type"=>"float", "doc_values"=>true}, "longitude"=>{"type"=>"float", "doc_values"=>true}}}}}}}, :level=>:info}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["localhost:9200"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
Pipeline main started

以下是启动ElasticSearch时收到的消息：

[2016-04-20 09:22:02,734][INFO ][node                     ] [Centurion] version[2.3.1], pid[7301], build[bd98092/2016-04-04T12:25:05Z]
[2016-04-20 09:22:02,735][INFO ][node                     ] [Centurion] initializing ...
[2016-04-20 09:22:03,394][INFO ][plugins                  ] [Centurion] modules [lang-groovy, reindex, lang-expression], plugins [], sites []
[2016-04-20 09:22:03,416][INFO ][env                      ] [Centurion] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [31.2gb], net total_space [49gb], spins? [no], types [ext4]
[2016-04-20 09:22:03,416][INFO ][env                      ] [Centurion] heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-04-20 09:22:03,416][WARN ][env                      ] [Centurion] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-04-20 09:22:05,313][INFO ][node                     ] [Centurion] initialized
[2016-04-20 09:22:05,313][INFO ][node                     ] [Centurion] starting ...
[2016-04-20 09:22:05,458][INFO ][transport                ] [Centurion] publish_address {127.0.0.1:9300}, bound_addresses {[::]:9300}
[2016-04-20 09:22:05,464][INFO ][discovery                ] [Centurion] elasticsearch/fSRusanNRgeBN9D38nXVFg
[2016-04-20 09:22:08,511][INFO ][cluster.service          ] [Centurion] new_master {Centurion}{fSRusanNRgeBN9D38nXVFg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-04-20 09:22:08,526][INFO ][http                     ] [Centurion] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[2016-04-20 09:22:08,526][INFO ][node                     ] [Centurion] started
[2016-04-20 09:22:08,588][INFO ][gateway                  ] [Centurion] recovered [2] indices into cluster_state
[2016-04-20 09:22:08,981][INFO ][cluster.routing.allocation] [Centurion] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibi][0]] ...]).

Answer 1

我建议将以下行添加到CSV文件的输入插件中。

sincedb_path => "/etc/logstash/sincedb_test.txt"

确保/etc/logstash/可写。

干杯。

Logstash不为csv文件类型创建索引

1 个答案: