Question

这是我的一个日志条目

INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E

这是我的模式，

DJANGOTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}

这是我的logstash conf文件，

input {
  beats {
    port => "5043"
 }
}
filter {
  if [type] in ["django"] {
    grok {
      patterns_dir => ["/opt/logstash/patterns"]
      match => [ "message" , "%{LOGLEVEL:level}%{SPACE}%{DJANGOTIMESTAMP:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:uid}%{SPACE}%{INT:django-id}%{SPACE}%{GREEDYDATA:action}" ]
    }
 }
 }
 output {
   elasticsearch {
    hosts => [ "localhost:9200" ]
    index => "%{type}_indexer"
 }
 }

在弹性搜索输出中，不会创建字段，

luvpreet@DHARI-Inspiron-3542:/usr/bin$ curl -XGET 'localhost:9200/django_indexer/_search?pretty=true&q=*:*'
      {
    "_index" : "django_indexer",
    "_type" : "django",
    "_id" : "AVwu8tE7j-Kh6vl1kUdf",
    "_score" : 1.0,
    "_source" : {
      "@timestamp" : "2017-05-22T06:55:52.819Z",
      "offset" : 144,
      "@version" : "1",
      "beat" : {
        "hostname" : "DHARI-Inspiron-3542",
        "name" : "DHARI-Inspiron-3542",
        "version" : "5.4.0"
      },
      "input_type" : "log",
      "host" : "DHARI-Inspiron-3542",
      "source" : "/var/log/django/a.log",
      "message" : "INFO 2017-05-16 06:33:08,673 views 40152 139731056719616 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E",
      "type" : "django",
      "tags" : [
        "beats_input_codec_plain_applied"
      ]
    }

这并不是说解析器失败了，但为什么这些字段没有被创建？我缺少什么？

Answer 1

尝试使用这个grok模式：

%{LOGLEVEL:loglevel}%{SPACE}%{TIMESTAMP_ISO8601:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:id}%{SPACE}%{INT:number}%{SPACE}%{URI:action}

输入

INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E

输出

number      139643033982720
timestamp   2017-05-16·17:24:11
id          14463
port    
pid         690
origin      views
action      https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E
loglevel    INFO

然后，您可以在过滤器插件中使用mutate删除端口字段

mutate {
    remove_field => ["port"]
}

<强>更新

好的，我在我的logstash上尝试了你的配置。这就是我所做的：

1-配置filebeat：

filebeat.prospectors:

- paths:
    - /etc/filebeat/FilebeatInputTest.txt
  document_type: django

output.logstash:
  hosts: ["127.0.0.1:5044"]

2-配置logstash

input {
  beats {
    port => "5044"
 }
}
filter {
  if [type] == "django" {
    grok {
      match => [ "message" , "%{LOGLEVEL:loglevel}%{SPACE}%{TIMESTAMP_ISO8601:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:id}%{SPACE}%{INT:number}%{SPACE}%{GREEDYDATA:action}" ]
    }
    mutate {
        remove_field => ["@timestamp", "beat","input_type","offset","source","@version","host","tags","message"]
    }
 }
}
output {
   elasticsearch {
    hosts => [ "xx.xx.xx.xx:9200" ]
    index => "%{type}_indexer"
    user => "xxxx"
    password => "xxxx"
  }
}

如果您的弹性搜索不受保护，则可以删除user和password。

输入（/etc/filebeat/FilebeatInputTest.txt的内容）

INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E

输出（弹性搜索）

{
    "_index" : "django_indexer",
    "_type" : "django",
    "_id" : "AVwhFe30JYGYNG_7C7YI",
    "_score" : 1.0,
    "_source" : {
        "origin" : "views",
        "pid" : "690",
        "type" : "django",
        "number" : "139643033982720",
        "loglevel" : "INFO",
        "action" : "https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E",
        "id" : "14463",
        "timestamp" : "2017-05-16 17:24:11"
    }
}

希望这有帮助。

grok解析器失败 - 对于django日志

1 个答案: