Question

在telegraf logparser中，我的配置段如下：

[[inputs.logparser]]
   files = ["/home/work/local/monitor/logs/xxx.log"]
   from_beginning = false
   watch_method = "inotify"
   [inputs.logparser.grok]
     patterns = ["%{LOG_LINE}"]
     measurement = "xxx_log"
     custom_pattern_files = ["/etc/telegraf/patterns_xxx.conf"]
     timezone = "UTC"

这样的日志

"a:b"
"c=d"

我的自定义模式：

PATTERN1 %{WORD:key}:%{WORD:value}
PATTERN2 %{WORD:key}=%{WORD:value}
LOG_LINE %{PATTERN1}|%{PATTERN2}

对于日志：

name=jack

LOG_LINE得到了

{"key": [["a",null]],"value": [["b",null]]}

但我想得到

{"key": ["a"],"value": ["b"]}

什么是正确的模式？谢谢！

Answer 1

您的过滤器配置如何？

我使用该示例测试了您的grok模式，它可以正常工作，我使用了以下过滤器。

filter {
    grok {
        patterns_dir => ["/etc/logstash/patterns/"]
        break_on_match => false
        match => ["message","%{LOG_LINE}"]
        tag_on_failure => [ "_grokparsefailure"]
    }
}

然后在目录/etc/logstash/patterns/中放入一个包含您的模式的文件。

PATTERN1 %{WORD:key}:%{WORD:value}
PATTERN2 %{WORD:key}=%{WORD:value}
LOG_LINE %{PATTERN1}|%{PATTERN2}

这是logstash输出。

{
  "@timestamp":"2018-07-13T14:29:25.180Z",
  "value":"d",
  "host":"logstash-lab",
  "message":"\"c=d\"",
  "key":"c",
  "@version":"1"
}
{
 "@timestamp":"2018-07-13T14:29:25.179Z",
 "value":"b",
 "host":"logstash-lab",
 "message":"\"a:b\"",
 "key":"a",
 "@version":"1"
}

Answer 2

/etc/telegraf/telegraf.conf

[[inputs.logparser]]
   files = ["/var/log/auth.log"]
   from_beginning = false
   watch_method = "inotify"
   [inputs.logparser.grok]
     patterns = ["%{LOG_LINE}"]
     measurement = "auth_log"
     custom_pattern_files = ["/home/local/conf.d/09-syslog-filter.conf"]
     timezone = "UTC"



cat /home/local/conf.d/09-syslog-filter.conf
filter {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
        pattern_definitions => {
          "GREEDYMULTILINE"=> "(.|\n)*"
        }
      }
      date {
        match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
      geoip {
        source => "[system][auth][ssh][ip]"
        target => "[system][auth][ssh][geoip]"
      }
LOG_LINE %{SYSLOGTIMESTAMP}|%{SYSLOGHOST}|%{POSINT}|%{DATA}
}



systemctl status telegraf.service
● telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB
   Loaded: loaded (/lib/systemd/system/telegraf.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Sun 2018-10-21 10:15:00 +06; 6min ago
     Docs: https://github.com/influxdata/telegraf
  Process: 30366 ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF
 Main PID: 30366 (code=exited, status=2)

Failed to start The plugin-driven server agent for reporting metrics into InfluxDB.


I need help.
[that grok is ok for logstash filter]

自定义grok模式不起作用

2 个答案: