Question

我尝试使用logstash将csv文件中的数据提供给弹性搜索。我的logsatsh配置文件如下所示：

input {
    file {
        path => "C:\Users\shreya\Data\RetailData.csv"
        start_position => "beginning" 
        #sincedb_path => "C:\Users\shreya\null"

    }
}
filter {
    csv {
        separator => ","
        id => "Store_ID"
        columns => ["Store","Date","Temperature","Fuel_Price", "MarkDown1", "MarkDown2", "MarkDown3", "MarkDown4", "CPI", "Unemployment", "IsHoliday"]
    }
    mutate {convert => ["Store", "integer"]}
    mutate {convert => ["Date", "date"]}
    mutate {convert => ["Temperature", "float"]}
    mutate {convert => ["Fuel_Price", "float"]}
    mutate {convert => ["CPI", "float"]}
    mutate {convert => ["Unemployment", "float"]}


}
output {
    elasticsearch {
        action => "index"
        hosts => "localhost:9200" 
        index => "store" 
        document_type => "store_retail"     
    }
    stdout {} 
    #stdout {
  #       codec => rubydebug
  #}
}

但我收到错误而无法找到解决问题的方法。我是logstash的新手。我的错误日志如下所示：

[2017-12-02T15：56：38,150] [INFO] [logstash.modules.scaffold]初始化模块{：module_name =＆gt;＆＃34; fb_apache＆＃34;，：directory =＆gt;＆＃34; C：/Users/shreya/logstash-6.0.0/modules/fb_apache/configuration"} [2017-12-02T15：56：38,165] [INFO] [logstash.modules.scaffold]初始化模块{：module_name =＆gt;＆＃34; netflow＆＃34;，：directory =＆gt;＆＃34; C：/用户/ SHREYA / logstash-6.0.0 /模块/网络流/配置＆＃34;} [2017-12-02T15：56：38,243] [警告] [logstash.config.source.multilocal]忽略＆＃39; pipelines.yml＆＃39; file，因为指定了模块或命令行选项 [2017-12-02T15：56：39,117] [INFO] [logstash.agent]已成功启动Logstash API端点{：port =＆gt; 9600} [2017-12-02T15：56：42,965] [警告] [logstash.outputs.elasticsearch]您正在使用已弃用的配置设置＆＃34; document_type＆＃34;在elasticsearch中设置。不推荐使用的设置将继续有效，但计划将来从logstash中删除。文档类型在Elasticsearch 6.0中已弃用，并在7.0中完全删除。您应该避免使用此功能如果您对此有任何疑问，请访问freenode irc上的#logstash频道。 {：name =＆gt;＆＃34; document_type＆＃34;，：plugin =＆gt;＆＃34; index＆＃34;，hosts =＆gt; [＆＃34; localhost：9200＆＃34;]，index =＆gt; ＆＃34; store＆＃34;，document_type =＆gt;＆＃34; store_retail＆＃34;，id =＆gt;＆＃34; 91a4406a13e9377abb312acf5f6be8e609a685f9c84a5906af957e956119798c＆＃34;＆gt;} [2017-12-02T15：56：43,704] [INFO] [logstash.outputs.elasticsearch] Elasticsearch池URL已更新{：changes =＆gt; {：removed =＆gt; []，：added =＆gt; [{{3} }}} [2017-12-02T15：56：43,804] [INFO] [logstash.outputs.elasticsearch]运行运行状况检查以查看Elasticsearch连接是否正常工作{：healthcheck_url =＆gt; http://localhost:9200/]，：path =＆gt;＆＃34; /＆＃34;} [2017-12-02T15：56：43,854] [警告] [logstash.outputs.elasticsearch]恢复与ES实例的连接{：url =＆gt;＆＃34; http://localhost:9200/＆＃34;} [2017-12-02T15：56：43,432] [INFO] [logstash.outputs.elasticsearch]使用{：path =＆gt; nil}中的映射模板 [2017-12-02T15：56：43,433] [INFO] [logstash.outputs.elasticsearch]试图安装模板{：manage_template =＆gt; {＆＃34;模板＆＃34; =＆gt;＆＃34; logstash- < em>＆＃34;，＆＃34;版本＆＃34; =＆gt; 60001，＆＃34;设置＆＃34; =＆gt; {＆＃34; index.refresh_interval＆＃34; =＆gt;＆＃34; 5s＆＃34;}，＆＃34;映射＆＃34; =＆gt; {＆＃34; _default _＆＃34; =＆gt; {＆＃34; dynamic_templates＆＃34; =＆gt; [{＆＃34; message_field＆＃34; =＆gt; {＆＃34; path_match＆＃34; =＆gt;＆＃34;消息＆＃34;，＆＃34; match_mapping_type＆＃34; =＆gt;＆＃34;字符串＆＃34;，＆＃34;映射＆＃ 34; =＆gt; {＆＃34;键入＆＃34; =＆gt;＆＃34;文字＆＃34;，＆＃34;规范＆＃34; =＆gt; false}}}，{＆＃34; string_fields＆＃34 ; =＆gt; {＆＃34;匹配＆＃34; =＆gt;＆＃34; ＆＃34;，＆＃34; match_mapping_type＆＃34; =＆gt;＆＃34;字符串＆＃34;，＆＃34;映射＆＃34; =＆gt; {＆＃34;键入＆＃34; =＆gt;＆＃34;文字＆＃34;，＆＃34;规范＆＃34; =＆gt; false，＆＃34;字段＆＃ 34; =＆gt; {＆＃34;关键字＆＃34; =＆gt; {＆＃34;类型＆＃34; =＆gt;＆＃34;关键字＆＃34;，＆＃34; ignore_above＆＃34; =＆gt; 256 }}，}}，＆＃34;属性＆＃34; =＆gt; {＆＃34; @ timestamp＆＃34; =＆gt; {＆＃34;类型＆＃34; =＆gt;＆＃34;日期＆＃34 ;}，＆＃34; @ version＆＃34; =＆g t; {＆＃34; type＆＃34; =＆gt;＆＃34; keyword＆＃34;}，＆＃34; geoip＆＃34; =＆gt; {＆＃34; dynamic＆＃34; =＆gt; true，＆＃34;属性＆＃34; =＆gt; {＆＃34; ip＆＃34; =＆gt; {＆＃34;类型＆＃34; =＆gt;＆＃34; ip＆＃34;}，＆＃34;位置＆＃ 34; =＆gt; {＆＃34;类型＆＃34; =＆gt;＆＃34; geo_point＆＃34;}，＆＃34;纬度＆＃34; =＆gt; {＆＃34;类型＆＃34; =＆gt; ＆＃34; half_float＆＃34;}，＆＃34;经度＆＃34; =＆gt; {＆＃34;类型＆＃34; =＆gt;＆＃34; half_float＆＃34;}}}}}}}}} [2017-12-02T15：56：43,964] [INFO] [logstash.outputs.elasticsearch]新Elasticsearch输出{：class =＆gt;＆＃34; LogStash :: Outputs :: ElasticSearch＆＃34;，：hosts =＆gt; [＆＃34; //本地主机：9200＆＃34;]} [2017-12-02T15：56：44,111] [错误] [logstash.pipeline]错误注册插件{：pipeline_id =＆gt;＆＃34; main＆＃34;，：plugin =＆gt;＆＃34;＃＆gt;， @ metric_events_time = org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter $ Proxy2 - 命名空间：[stats，pipelines，main，plugins，filters，e3501f879986420bd95a59d8a1c006d9bc4351a481c96bd5366e7edb54bc6fbb，events] key：duration_in_millis value：0，@ id = \＆＃34; e3501f879986420bd95a59d8a1c006d9bc4351a481c96bd5366e7edb54bc6fbb \＆＃34;，@ klass = LogStash :: Filters :: Mutate，@ metric_evers =＃，@ structured_lookup_mutex =＃，@ fast_lookup =＃＆gt;＆gt;＆gt;，@ namespace_name = [：stats ，：pipelines，：main，：plugins，：filters，：e3501f879986420bd95a59d8a1c006d9bc4351a481c96bd5366e7edb54bc6fbb，：events]＆gt;，@ filter = {\＆＃34; Date \＆＃34; =＆gt; \＆＃34; date \＆＃34 ;}，id =＆gt; \＆＃34; e3501f879986420bd95a59d8a1c006d9bc4351a481c96bd5366e7edb54bc6fbb \＆＃34;，enable_metric =＆gt; true，periodic_flush =＆gt; false＆gt;＆gt;＆＃34;，：error =＆gt;＆＃34;翻译缺失： en.logstash.agent.configuratio n.invalid_plugin_register＆＃34;，：thread =＆gt;＆＃34;＃＆＃34;} [2017-12-02T15：56：44,142] [错误] [logstash.pipeline]由于错误导致管道中止{：pipeline_id =＆gt;＆＃34; main＆＃34;，：exception =＆gt;＃，：backtrace =＆gt [＆＃34; C：/Users/shreya/logstash-6.0.0/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.1.6/lib/logstash/filters/mutate.rb ：186：block in register'", "org/jruby/RubyHash.java:1343:in每个＆＃39;＆＃34;，＆＃34; C：/Users/shreya/logstash-6.0.0/vendor/bundle/jruby/2.3.0/gems/logstash-filter -mutate-3.1.6 / lib / logstash / filters / mutate.rb：184：在register'", "C:/Users/shreya/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:388:in register_plugin＆＃39;＆＃34;，＆＃34; C：/Users/shreya/logstash-6.0.0 /logstash-core/lib/logstash/pipeline.rb:399:in block in register_plugins'", "org/jruby/RubyArray.java:1734:in每个＆＃39;＆＃34;，＆＃34; C：/Users/shreya/logstash-6.0.0/logstash-core /lib/logstash/pipeline.rb:399:in register_plugins'", "C:/Users/shreya/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:801:in maybe_setup_out_plugins＆＃39;＆＃34;，＆＃34; C：/Users/shreya/logstash-6.0.0/logstash-core/lib/logstash /pipeline.rb:409:in start_workers'", "C:/Users/shreya/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:333:in运行＆＃39;＆＃34;，＆＃34; C：/Users/shreya/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb ：293：在`block in start＆＃39;＆＃34;]中，：thread =＆gt;＆＃34;＃＆＃34;} [2017-12-02T15：56：44,158] [错误] [logstash.agent]无法执行操作{：id =＆gt;：main，：action_type =＆gt; LogStash :: ConvergeResult :: FailedAction，：message =＆gt; ＆＃34;无法执行操作：LogStash :: PipelineAction :: Create / pipeline_id：main，action_result：false＆＃34;，：backtrace =＆gt; nil}

Answer 1

问题来自其中一个mutate过滤器中的转换目标。来自documentation：

有效转换目标是：integer，float，string和boolean。

所以这部分导致了崩溃：

mutate {convert => ["Date", "date"]}

如果您想要将String转换为日期，则必须使用日期过滤器。

Answer 2

使用以下命令验证您的配置文件，该命令显示错误详细信息。

./logstash -f /etc/logstash/conf.d/your_config_file.conf --config.test_and_exit

logstash配置文件错误

2 个答案: