使用Logstash进行弹性搜索的逻辑 - 简单配置

时间：2014-10-18 08:54:38

标签： elasticsearch logstash kibana logary

我一直在尝试使用Logary＆gt;为我的.NET应用程序设置日志记录。 Logstash＆gt; ElasticSearch几乎整整一天。我已经尝试了我能想到的配置的每个变体，并经过几个小时的文档无济于事。

这里有什么问题？

 var logary = LogaryFactory.New("Logary Specs", with => with

                .Target<Logstash.Builder>("ls", conf => conf.Target
                        .Hostname("localhost")
                        .Port(1939)
                        .EventVersion(Logstash.EventVersion.One)
                        .Done())

                .Target<Debugger.Builder>("debugger")
            );

这是logstash.conf：

input { 
    stdin {}

    tcp {
         codec => json_lines { charset => "UTF-8" }
         host  => localhost
         port  => 1939
         type  => "app"
     }
}

output {

   stdout {}
   elasticsearch {
      host => localhost
      protocol => "http"
   }
}

我试过没有output.elasticsearch.protocol和host =＆gt;像文档中的“0.0.0.0”，以及介于两者之间的每个组合。

我已经在9200上升了ES并且Kibana加载了，我可以看到日志通过stdin / stdout显示，但不是来自使用Logary的应用程序日志。它还验证了Logary调试器目标的工作原理。

我的管道出了什么问题？

Logstash 1.4.2

ES 1.3.4

2 个答案:

答案 0 :(得分：1)

您可以尝试一些事情。首先仔细检查您的elasticsearch簇是否实际为绿色/黄色。我浪费了几个小时一次，因为我在我配置的elasticsearch索引模板中指定了更多的副本。

当然，检查elasticsearch和logstash日志是否有任何错误。

接下来，尝试使用

elasticsearch_http { host => "localhost" }

这使用http协议而不是内部协议，并且释放您必须具有与logstash捆绑的完全相同的弹性搜索版本。

最后，如果这不起作用，我建议在logstash目录中浏览并修改实现插件的ruby脚本。当它工作时它很好但是当它没有时，它有点缺乏记录;这可以通过一些日志语句轻松修复。

答案 1 :(得分：1)

您需要一些组件来调试它。

我写了一个示例程序供您用来调试它;这是最简单的可行方法：

open System

open Logary
open Logary.Configuration
open Logary.Targets

[<EntryPoint>]
let main argv = 
  use logary =
    withLogary' "logstash-sample" (
      withTargets [
        Logstash.create (Logstash.LogstashConf.Create "10.0.0.120") "logstash"
        Debugger.create Debugger.empty "debugger"
      ] >>
      withRules [
        Rule.createForTarget "logstash"
        Rule.createForTarget "debugger"
      ] >>
      withInternalTargets Verbose [
        Debugger.create (Debugger.empty) "console"
      ]
    )

  let logger = logary.GetLogger "logstash-sample"
  Logger.debug logger "Hello World"

  Console.ReadKey true |> ignore
  0

这是预期的调试器输出（不是控制台）。

'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\logstash_sample.exe', Symbols loaded.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Logary.dll', Symbols loaded.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\NodaTime.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\FSharp.Core\v4.0_4.3.0.0__b03f5f7f11d50a3a\FSharp.Core.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Xml\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Newtonsoft.Json.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Intelliplan.JsonNet.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Intelliplan.JsonNet.NodaTime.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\FSharp.Actor.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
D 2014-10-18T10:25:42.4192736+00:00: supervisor status: Running [Logary.Registry.runRegistry]
I 2014-10-18T10:25:42.4192736+00:00: initing logstash target, connecting to 10.0.0.120:1936... [Logary.Targets.Logstash.logstashLoop]
D 2014-10-18T10:25:42.4383230+00:00: scheduler status: Running [Logary.Registry.runRegistry]
D 2014-10-18T10:25:42.4473276+00:00: registry status: Running [Logary.Registry.runRegistry]
D 2014-10-18T10:25:42.4977127+00:00: will poll metrics every 0:00:00:00.5 [Logary.Registry.registry]
D 2014-10-18T10:25:42.5335455+00:00: Hello World [logstash-sample]
I 2014-10-18T10:25:42.5515369+00:00: initing logstash target, connected [Logary.Targets.Logstash.logstashLoop]
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Numerics\v4.0_4.0.0.0__b77a5c561934e089\System.Numerics.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.ComponentModel.DataAnnotations\v4.0_4.0.0.0__31bf3856ad364e35\System.ComponentModel.DataAnnotations.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Runtime.Serialization\v4.0_4.0.0.0__b77a5c561934e089\System.Runtime.Serialization.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Xml.Linq\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.Linq.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
D 2014-10-18T10:25:42.7268749+00:00: polling metrics [Logary.Registry.registry]
D 2014-10-18T10:25:42.8571421+00:00: send-recv stream not open, opening [Logary.Targets.Logstash.logstashLoop]
I 2014-10-18T10:25:43.1096339+00:00: start shutdown [Logary.Configuration.Config.shutdown]
I 2014-10-18T10:25:43.1186538+00:00: flush start [Logary.Registry.registry]
I 2014-10-18T10:25:43.1667422+00:00: flush Ack [Logary.Registry.registry]
I 2014-10-18T10:25:43.1787861+00:00: shutdown metrics polling [Logary.Registry.registry]
I 2014-10-18T10:25:43.1798152+00:00: shutdown schedules [Logary.Registry.registry]
I 2014-10-18T10:25:43.1798152+00:00: shutdown targets [Logary.Registry.registry]
I 2014-10-18T10:25:43.2018104+00:00: shutdown Ack [Logary.Registry.registry]
I 2014-10-18T10:25:43.2018104+00:00: shutting down immediately [Logary.Registry.registry]
I 2014-10-18T10:25:43.2118258+00:00: stop shutdown [Logary.Configuration.Config.shutdown]
The program '[4820] logstash_sample.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).

此代码在此处：

https://github.com/logary/logary/commit/0dca450677dc1ca691292d379a734c7051743c1c

您可以执行Install-Package Intelliplan.Logary -Pre，然后复制Logary.dll - dll文件。确保您具有相同的输出。

如果您可以按照上面的输出成功连接到logstash，则可以通过打印到其日志来验证它是否正在接收事件或失败; 

tail -f /var/log/logstash/logstash-agent.log
相关问题
最新问题