Question

我是Logstash和grok的新手，我试图以以下格式解析S3存储桶中的AWS ECS日志-

文件名-my-logs-s3-bucket/3d265ee3-d2ee-4029-a3d9-fd2255d69b92/ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842/000000.gz

示例行-

2019-05-09T16:16:16.983Z   JBoss Bootstrap Environment
2019-05-09T16:16:16.983Z   JBOSS_HOME: /app/jboss
2019-05-09T16:16:16.983Z   JAVA_OPTS:  -server -XX:+UseCompressedOops  -Djboss.server.log.dir=/var/log/jboss -Xms128m -Xmx4096m

和logstash.conf

input {
  s3 {
    region => "us-east-1"
    bucket => "my-logs-s3-bucket"
    interval => "7200"
  }
}

filter {
  grok {
    match => ["message", "%{TIMESTAMP_ISO8601:tstamp}"]
  }
  date {
    match => ["tstamp", "ISO8601"]
  }
  mutate {
    remove_field => ["tstamp"]
    add_field => {
      "file" => "%{[@metadata][s3][key]}"
    }
    ######### NEED HELP HERE - START #########
    #grok {
    #  match => [ "file", "ecs-fargate-container-%{DATA:containerlogname}"]
    #}
    ######### NEED HELP HERE - END #########
  }
}

output {
 stdout { codec => rubydebug {
    #metadata => true
  }
 }
}

当我使用上述配置运行logstash时，我能够看到所有已解析的日志和提取的文件名，并且输出中的文件名如下-

"file" => "myapp-logs/3d265ee3-d2ee-4029-a3d9-fd2255d69b92/ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842/000000.gz",

我试图通过取消注释ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842之间的grok配置行并以以下错误结尾-

来使用grok将文件名提取为8ff0e472-c76f-4f61-a363-64c2b80aa842或#NEED HELP HERE - START。

Expected one of #, => at line 21, column 10 (byte 536) after filter {\n grok {\n match => [\"message\", \"%{TIMESTAMP_ISO8601:tstamp}\"]\n }\n date {\n match => [\"tstamp\", \"ISO8601\"]\n }\n mutate {\n #remove_field => [\"tstamp\"]\n add_field => {\n \"file\" => \"%{[@metadata][s3][key]}\"\n }\n grok ", :

我不确定这是哪里出了问题。请指教。

Answer 1

您的grok过滤器位于mutate过滤器内部，请尝试以下操作。

filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:tstamp}"]
    }
    date {
        match => ["tstamp", "ISO8601"]
    }
    mutate {
        remove_field => ["tstamp"]
        add_field => { "file" => "%{[@metadata][s3][key]}" }
    }
    grok {
        match => [ "file", "ecs-fargate-container-%{DATA:containerlogname}"]
    }
}

Logstash grok配置错误-解析文件名时

1 个答案: