Question

在提取节点Pipeline中解析日期和时间的问题。

解析Cloudfront日志时间戳的问题：

日志示例：

2019-01-23  00:47:29    FRA2    542 113.139.67.32   GET xxxxx.net   /media/no_image 404 -   okhttp/3.8.1    -   -   Error   ODvvwZI_z12W-4En6l_uVxxxxxxxHy9fA== xxxx.net    https   452 0.051   -   TLSv1.2 ECDHE-sdsCM-SHA256  Error   HTTP/2.0    -   -

管道：

 "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{DATE_EU:date}\t%{TIME:time}\t%{GREEDYDATA:x_edge_location}\t(?:%{NUMBER:sc_bytes:int}|-)\t%{IPORHOST:clinetip}\t%{WORD:cs_method}\t%{HOSTNAME:cs_host}\t%{NOTSPACE:cs_uri_stem}\t%{NUMBER:sc_status:int}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:User_Agent}\t%{GREEDYDATA:cs-uri-query}\t%{GREEDYDATA:cookies}\t%{WORD:x_edge_result_type}\t%{NOTSPACE:x_edge_request_id}\t%{HOSTNAME:x_host_header}\t%{URIPROTO:cs_protocol}\t%{INT:cs_bytes:int}\t%{NUMBER:time_taken:float}\t%{GREEDYDATA:x_forwarded_for}\t%{GREEDYDATA:ssl_protocol}\t%{GREEDYDATA:ssl_cipher}\t%{GREEDYDATA:x_edge_response_result_type}\t%{GREEDYDATA:cs-protocol-version}\t%{GREEDYDATA:fle-status}\t%{GREEDYDATA:fle-encrypted-fields}"]
      }
    },
    {
      "set": {
        "field": "listener_timestamp",
        "value": "{{date}} {{time}}"
      }
    },
    {
      "date": {
        "field": "listener_timestamp",
        "target_field": "@timestamp",
        "formats": [
          "yy-MM-dd HH:mm:ss"
        ]
      }
    },
]

在json中：

"@timestamp": "2019-01-23T00:00:00.000Z"

在logstash配置中，如下所示：

filter {
  grok {
    match => { "message" => "%{DATE_EU:date}\t%{TIME:time}\t%{WORD:x_edge_location}\t(?:%{NUMBER:sc_bytes:int}|-)\t%{IPORHOST:c_ip}\t%{WORD:cs_method}\t%{HOSTNAME:cs_host}\t%{NOTSPACE:cs_uri_stem}\t%{NUMBER:sc_status:int}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:User_Agent}\t%{GREEDYDATA:cs_uri_stem}\t%{GREEDYDATA:cookies}\t%{WORD:x_edge_result_type}\t%{NOTSPACE:x_edge_request_id}\t%{HOSTNAME:x_host_header}\t%{URIPROTO:cs_protocol}\t%{INT:cs_bytes:int}\t%{GREEDYDATA:time_taken}\t%{GREEDYDATA:x_forwarded_for}\t%{GREEDYDATA:ssl_protocol}\t%{GREEDYDATA:ssl_cipher}\t%{GREEDYDATA:x_edge_response_result_type}" }
  }

  mutate {
    add_field => [ "listener_timestamp", "%{date} %{time}" ]
  }

  date {
    match => [ "listener_timestamp", "yy-MM-dd HH:mm:ss" ]
    target => "@timestamp"
  }
}

这里是将Logstash配置转换为摄取管道配置的任何转换器？

Answer 1

问题已解决：

{
      "set" : {
        "field": "listener_timestamp",
        "value": "{{date}} {{time}}"
      }
    },
    {
      "date": {
        "field": "listener_timestamp",
        "target_field": "@timestamp",
        "formats": [
          "yy-MM-dd HH:mm:ss"
        ]
      }
    },

用于解析Cloudfront日志的管道

1 个答案: