Question

我在使用logstash将CSV文件导入ElasticSearch以在Kibana中进行进一步处理时遇到问题。

这是我的logstash配置文件：

input {
    file {
        path => ["/absolute_path_to_file/export.csv"]
        start_position => beginning 
        ignore_older => 0 
        sincedb_path => "/dev/null"
    }
}
#filter {
#   csv {
#       columns => [
#           "id",
#           "cislo_smlouvy",
#           "zdroj",
#           "produkt",
#           "sjednani",
#           "datum_odeslani",
#           "cas_odeslani",
#           "pojistovna",
#           "tarif",
#           "pojistnik",
#           "telefon",
#           "predmet_pojisteni",
#           "rz",
#           "pocatek_pojisteni",
#           "rocni_pojistne",
#           "urgence",
#           "stav"
#       ]
#       separator => ";"
#       remove_field => ["message"]
#   }
#}
output {
#   elasticsearch {
#       hosts => "localhost:9200"
#       index => "smlouvy"
#   }
    stdout {
        codec => rubydebug
    }
}

我的CSV文件摘录：

"id";"číslo smlouvy";"zdroj";"produkt";"sjednání";"datum odeslaní";"čas odeslání";"pojišťovna";"tarif";"pojistník";"pojistnik telefon";"předmět pojištění";"rz";"počátek";"roční pojistné";"urgence";"stav"
"114951";"6132681255";"SRO";"POV";;"1.6.2016";"12:28";"csob";"csob-2";"BB TEST";"721666333";"Škoda Favorit";"NENÍ";"2.6.2016 00:00";"4657,00";;"TEST"
"114950";;"POV";"POV";"VO Bukvicova";"1.6.2016";"12:16";"csob";"csob-2";"BB BB";"721000111";"BMW X3";"NENÍ";"3.6.2016 00:00";"5550,00";;"TEST"

我叫这个命令：
sudo logstash -f /absolute_path_to_file/logstash.conf --vebrose

使用以下输出：

starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 2
Registering file input {:path=>["/absolute_path_to_file/export.csv"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
Pipeline main started

经过一段时间无所事事，我把它关闭了：

^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Closing inputs {:level=>:info}
Closed inputs {:level=>:info}
Input plugins stopped! Will shutdown filter/output workers. {:level=>:info}
Pipeline main has been shutdown

可能相关的版本信息：

logstash 2.3.2
logstash-input-file (2.2.5)
logstash-filter-csv (2.1.3)
logstash-output-elasticsearch (2.6.2)
logstash-output-stdout (2.0.6)
logstash-codec-rubydebug (2.0.7)

我已经阅读了所有可以找到的文档，并尝试从GitHub复制了很多logstash.conf示例，但没有运气。对我失踪的任何帮助？

Answer 1

所以我终于找到了问题所在。这是输入CSV文件。

对于新行，CSV文件的\r为\n，而logstash默认为\r。

BTW：您无法将\n设置为logstash文件输入过滤器配置中的分隔符，因此我必须将CSV文件转换为{{1}}

的文件

无法使用Logstash

1 个答案: