Question

我想使用Logstash分离同一目录中的日志，以指向kibana仪表板中的唯一索引。我创建了一个手动索引，如下所示：

   #- input_type: log

  #paths:
   # /scratch/rmbbuild/logs/threadpool/*

  #multiline.pattern: '^[[:space:]]*-[[:space:]]*[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'

  #multiline.negate: true

  #multiline.match: after

- input_type: log

  paths:
    /scratch/rmbbuild/logs/threadpoolworkerstderr/*.stderr

  multiline.pattern: '^[[:space:]]*[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'

  multiline.negate: true

  multiline.match: after

  fields:
        mycustomvar: production

在同一线程池目录中有两种类型的日志，我想创建两个索引，一个用于线程池，另一个用于线程池workerstderr。并且要确保将此日志堆叠并存储在使用Logstash创建的各个索引中。 Logstash代码如下：

if [source]=~/threadpoolworkerstderr*.*/ {
 if [mycustomvar] == "production" {
grok {
        match => { "message" => "%{SPACE}%{TIME:timestamp} %{DATA:thread} %{LOGLEVEL:Loglevel} +\(%{DATA:class}\) (?<errormessage>(.|\r|\n)*)"}
}
mutate {
    replace => {"type" => "threadpoolstderr"}
}
}
}

if [source]=~/threadpoolworkerstderr*.*/ {
if [mycustomvar] == "production" {
grok {
        match => {"source" => "%{GREEDYDATA}/%{GREEDYDATA:batchname}\.(?<tempsdatestamp>([0-9]*))\.%{GREEDYDATA}\.%{GREEDYDATA:threadnumber}\.stderr"}
}
mutate {
    replace => {"type" => "threadpoolstderrinfo"}
}
}
}


if [source]=~/threadpoolworkerstderr*.*/ {
if [mycustomvar] == "production" {
grok {
        match => {"tempsdatestamp" => "(?<year>([0-9]{4}))(?<month>([0-9]{2}))(?<day>([0-9]{2}))"}
}
mutate {
    replace => {"type" => "threadpoolstderrinfo2"}
}
}
}


if [source]=~/threadpoolworkerstderr*.*/ {
if [mycustomvar] == "production" {

mutate{
    add_field => ["datestamp", "%{year}-%{month}-%{day}"]
    remove_field => ["tempsdatestamp", "year", "month", "day"]
    replace => {"type" => "threadpoolstderrinfo3"}
}
}
}
if [source]=~/threadpoolworker*.*/ {
grok {
        match => { "message" => "%{SPACE}-%{SPACE}%{TIMESTAMP_ISO8601:datetime} %{DATA:thread} %{LOGLEVEL:Loglevel} +\(%{DATA:class}\) %{GREEDYDATA:messagetext}" }
}
mutate {
    replace => { "type" => "threadpoolworkerlog" }
}
}

if [source]=~/threadpoolworker*.*/ {
grok {
        match => {"source" => "%{GREEDYDATA}/%{GREEDYDATA}/%{GREEDYDATA}/%{GREEDYDATA}/%{GREEDYDATA:ORMB_VERSION}/%{GREEDYDATA}"}
}
mutate {
    replace => { "type" => "threadpoolworkerlog1" }
}
}


output {

if [type] == "accesslogs1" {
 elasticsearch {
        hosts => [ "localhost:9200" ]
            user => logstash_internal
        password => changeme
        index => "accesslogindex"
    }
} 
else if [type] == "threadpoolstderrinfo3" {
 elasticsearch {
        hosts => [ "localhost:9200" ]
        user => logstash_internal
        password => changeme
        index => "threadpoolstderrindex"
   }
} else if [type] == "threadpoolstderrinfo2" {
 elasticsearch {
        hosts => [ "localhost:9200" ]
        user => logstash_internal
        password => changeme
        index => "threadpoolstderrindex"
   }
} else if [type] == "threadpoolstderrinfo" {
 elasticsearch {
        hosts => [ "localhost:9200" ]
        user => logstash_internal
        password => changeme
        index => "threadpoolstderrindex"
    }
} else if [type] == "threadpoolstderr" {
 elasticsearch {
        hosts => [ "localhost:9200" ]
        user => logstash_internal
        password => changeme
        index => "threadpoolstderrindex"
    }
} 
 else if [type] == "threadpoolworkerlog1" {
 elasticsearch {
        hosts => [ "localhost:9200" ]
           user => logstash_internal
           password => changeme
           index => "threadpoolworkerlogindex"
   }
} else if [type] == "threadpoolworkerlog" {
elasticsearch {
           hosts => [ "localhost:9200" ]
            user => logstash_internal
           password => changeme
           index => "threadpoolworkerlogindex"
    }
}

但是我看不到在Kibana仪表板上的threadpoolworkererr索引中堆积了threadpoolerr日志。但是我却只将这些日志存储在threadpoolworker索引中。因此，解决该问题的任何帮助将不胜感激。

如何过滤掉同一目录内的日志，以便它们只能指向某些特定索引使用ELK

0 个答案: