Question

我想截断/修剪我的logstash配置。

如您所见，它很长，总是重复同样的操作。我无法最小化/消除代码/配置双重化。

请参见下文：对于每个tomcat实例，都有应用程序，访问，标准输出和标准错误日志，这些日志通过filebeat传送到elkserver：9200。

大约有15个实例，最终有60条“ if”和“ else”指令。

有人提示如何截断/修剪输出节吗？

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/root-ca.pem"]
    ssl_certificate => "/etc/logstash/elkserver.pem"
    ssl_key => "/etc/logstash/elkserver.key"
    ssl_verify_mode => "force_peer"
  }
}

filter {
  grok { match => ["message",'^%{TIMESTAMP_ISO8601:TIMESTAMP} %{LOGLEVEL:LEVEL} *\[(?<CLASS>[A-Za-z0-9$]+).%{NOTSPACE:METHOD}:%{NONNEGINT:LINE}:%{NOTSPACE:THREAD}\] %{GREEDYDATA:MESSAGE}$'] }
  if "beats_input_codec_plain_applied" in [tags] { mutate { remove_tag => ["beats_input_codec_plain_applied"] } }
}

output {
  if "jt09_02_access" in [tags] {
    elasticsearch {
      hosts => ["elkserver:9200"]
      user => user
      password => password
      ssl => true
      ssl_certificate_verification => true
      cacert => "/etc/logstash/master.pem"
      index => "jt09_02_access"
    }
  } else if "jt07_02_access" in [tags] {
      elasticsearch {
        hosts => ["elkserver:9200"]
        user => user
        password => password
        ssl => true
        ssl_certificate_verification => true
        cacert => "/etc/logstash/master.pem"
        index => "jt07_02_access"
      }
  } else if "jt07_04_access" in [tags] {
      elasticsearch {
        hosts => ["elkserver:9200"]
        user => user
        password => password
        ssl => true
        ssl_certificate_verification => true
        cacert => "/etc/logstash/master.pem"
        index => "jt07_04_access"
      }
  } else if "jt07_01_access" in [tags] {
      elasticsearch {
        hosts => ["elkserver:9200"]
        user => user
        password => password
        ssl => true
        ssl_certificate_verification => true
        cacert => "/etc/logstash/master.pem"
        index => "jt07_01_access"
      }
  }  else if "jt07_09_sdterr" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         ssl_certificate_verification => true
         cacert => "/etc/logstash/master.pem"
         index => "jt07_09_sdterr"
       }
  }  else if "jt07_09_sdtout" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         ssl_certificate_verification => true
         cacert => "/etc/logstash/master.pem"
         index => "jt07_09_sdtout"
       }
  }  else if "jt07_09_custom_pattern" in [tags] {
        elasticsearch {
          hosts => ["elkserver:9200"]
          user => user
          password => password
          ssl => true
          ssl_certificate_verification => true
          cacert => "/etc/logstash/master.pem"
          index => "jt07_09_custom_pattern"
        }
  }  else if "jt09_01_access" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt09_01_access"
       }
  }  else if "jt09_03_access" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt09_03_access"
       }
  }  else if "jt09_01_sdterr" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt09_01_sdterr"
       }
  }  else if "jt09_01_sdtout" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt09_01_sdterr"
       }
  }  else if "jt09_02_sdterr" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt09_02_sdterr"
       }
  }  else if "jt09_02_sdtout" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt09_02_sdtout"
       }
  }  else if "jt09_03_sdterr" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt09_03_sdterr"
       }
  }  else if "jt09_03_sdtout" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt09_03_sdtout"
       }
  }  else if "jt08_03_access" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt08_03_access"
       }
  }  else if "jt08_03_sdterr" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt08_03_sdterr"
       }
  }  else if "jt08_03_sdtout" in [tags] {
       elasticsearch {
         hosts => ["elkserver:9200"]
         user => user
         password => password
         ssl => true
         cacert => "/etc/logstash/master.pem"
         index => "jt08_03_sdtout"
       }
  }
}

感谢Michael Dz，您的回答帮助我通过以下方式解决了我的问题：

Filebeat在发送数据时已经添加了必需的标签

- type: log
  paths: pathto\log_custompattern.log
  tags: ["jt07_09_custom_pattern"]
  close_older: 24h

- type: log
  paths: pathto\tomcat-stdout.??????????.log
  tags: ["jt07_09_sdtout"]
  scan_frequency: 30s
  close_inactive: 12h

- type: log
  paths: pathto\tomcat-stderr.??????????.log
  tags: ["jt07_09_sdterr"]
  scan_frequency: 30s
  close_inactive: 12h
  ...

Logstash现在以以下方式进行输出

output {
  elasticsearch {
    hosts => ["ma-javainfra02.konzern.mvvcorp.de:9200"]
    user => admin
    password => logfileselkadmin
    ssl => true
    ssl_certificate_verification => true
    cacert => "/etc/logstash/master.pem"
    index => "%{tags[0]}"
  }
}

Answer 1

您可以将标签添加为新字段，然后基于新的字段名称创建索引，我不知道您如何制作标签或在数组中存储了多少标签，但我假设您是对第一个感兴趣。

@Table(name = "users")
public class User {
    ...
    @OneToOne
    @JoinColumn(name = "address_id", insertable = false, updatable = false)
    private Address address;
    ...
}



@Table(name = "companies")
public class Company {
    ...
    @OneToOne
    @JoinColumn(name = "address_id", insertable = false, updatable = false)
    private Address address;
    ...
}

截断/修剪logstash配置易于维护

1 个答案: