Question

在此先感谢您的帮助！

我正在使用Netdata从服务器收集指标，然后将其发送到Logstash和Elastic。

我需要使用相同的字段来汇总指标，并以嵌套格式创建单个事件。

这是来自Netdata的输入示例：

{"host":"centosdns","@version":"1","port":52212,"@timestamp":"2019-01-19T16:16:22.117Z","message":"netdata.centosdns.disk_await.centos_swap.reads 0.0000000 1547914548"}
{"host":"centosdns","@version":"1","port":52212,"@timestamp":"2019-01-19T16:16:22.117Z","message":"netdata.centosdns.disk_await.centos_swap.writes 0.0000000 1547914548"}
{"host":"centosdns","@version":"1","port":52212,"@timestamp":"2019-01-19T16:16:22.117Z","message":"netdata.centosdns.disk_await.centos_root.reads 0.0000000 1547914548"}
{"host":"centosdns","@version":"1","port":52212,"@timestamp":"2019-01-19T16:16:22.117Z","message":"netdata.centosdns.disk_await.centos_root.writes 0.0000000 1547914548"}

logstash的配置文件如下：

input {
    tcp {
      port => 1234
    }
}

filter {
    # I take 'message' field and separate in different fields
    grok {
      named_captures_only => "true"
      pattern_definitions => {
        "CHART" => "[a-z]\w+"
        "FAMILY" => "[_a-z0-9]+"
    }

    match => {
      "message" => "%{WORD:prefix}\.%{WORD:hostname}\.%{CHART:chart}\.%{FAMILY:family}\.%{NOTSPACE:dimension} %{NUMBER:val} %{NUMBER:timestamp}" 
    }
}

if "_grokparsefailure" not in [tags] {
    mutate {
      remove_field => [ "@version", "host", "port", "prefix" ]
    }

    # Attempt to create a nested field and then aggregate 
    mutate {
      id => "chart_field"
      add_field => { "[%{chart}][%{family}][%{dimension}][value]" => "%{val}"
      }
    }

    aggregate {
      task_id => "[%{chart}][%{family}]"
      code => "
      # I tried many codes to aggregate but without success 
      event.cancel()
      "
      push_previous_map_as_event => true
      timeout => 5
    }

    mutate {
      # Remove unnecessary fields
      id => "netdata_mutate_remove"
      remove_field => [ "timestamp", "message"]
    }
} else {
    drop{}
}

output {
# TESTING PURPOSES
if "_aggregateexception" in [tags] {

    file {
      path => "/var/log/logstash/netdata/aggregatefailures-%{+MM-dd}.log"
    }
} else {
    file {
      path => "/var/log/logstash/netdata/netdata-%{+MM-dd}-aggregate.log"
    }
}

stdout { codec => rubydebug }
}

接受上面的输入：

"netdata.centosdns.disk_await.centos_swap.reads 0.0000000"
"netdata.centosdns.disk_await.centos_swap.writes 0.0000000"

我的目标是创建一个嵌套字段，例如：

disk_await: { # Chart
  centos_swap: { # Family
   [
    reads => 0.0000000, # Dimension => Value
    writes => 0.0000000 # Dimension => Value
   ]
  }
}

我假装将所有“ Dimension \'Value”汇总到同一个“ Chart” \“ Family”中，这只是四行指标，但实际上，我们谈论的是每秒1000条甚至在某些情况下甚至更多指标是动态的，几乎不可能知道所有名称。

此刻我正在使用：

Logstash v.6.5.4 on a Virtualbox CentOS 7 minimal
All plugins (inputs/filters/outputs) updated

如何从Logstash中的NetData汇总指标？

0 个答案: