Question

我有以下两行粘贴的日志文件：

11月26日14:20:32 172.16.0.1 date = 2014-11-26 time = 14:18:37 devname = XXXXCCCFFFFF devid = XXXCCVVGFFDD logid = 3454363464 type = traffic subtype = forward level = notice vd = root srcip = 172.16.1.251 srcport = 62032 srcintf =＆＃34; Combo_LAN＆＃34; dstip = X.X.X.X dstport = X dstintf =＆＃34; wan2＆＃34; sessionid = 16172588 status = close user =＆＃34; X.X＆＃34; group =＆＃34; Open Group＆＃34; policyid = 2 dstcountry =＆＃34;美国＆＃34; srccountry =＆＃34;保留＆＃34; trandisp = snat transip = X.X.X.X transport = X service = HTTP proto = 6 applist =＆＃34; Block_Applications＆＃34; duration = 11 sentbyte = 2377 rcvdbyte = 784 sentpkt = 6 rcvdpkt = 7 identidx = 5 utmaction = passthrough utmevent = webfilter utmsubtype = ftgd-cat urlcnt = 1 hostname =＆＃34; tacoda.at.atwola.com＆＃34; catdesc =＆＃34;广告＆＃34;

11月26日14:20:32 172.16.0.1 date = 2014-11-26 time = 14:18:37 devname = XXXXCCCFFFFF devid = XXXCCVVGFFDD logid = 3454363464 type = utm subtype = webfilter eventtype = ftgd_allow level = notice vd = ＆＃34;根＆＃34; policyid = 2 identidx = 5 sessionid = 15536743 user =＆＃34; X.X＆＃34; srcip = X.X.X.X srcport = X srcintf =＆＃34; Combo_LAN＆＃34; dstip = X.X.X.X dstport = 80 dstintf =＆＃34; wan2＆＃34;服务=＆＃34; HTTP＆＃34;主机名=＆＃34; streaming.sbismart.com＆＃34; profiletype =＆＃34; Webfilter_Profile＆＃34; profile =＆＃34;打开Group_Policy＆＃34;状态=＆＃34;直通＆＃34; reqtype =＆＃34;直接＆＃34; URL =＆＃34; /扩散/＆＃34; sentbyte = 984 rcvdbyte = 202 msg =＆＃34; URL属于策略中允许的类别＆＃34; method = domain class = 0 cat = 18 catdesc =＆＃34;经纪和交易＆＃34;

我的问题是，如果列数和顺序是固定的，我可以解析数据。

但是，我如何解析配置文件中的动态列，以便我不能获得_grokparsefailure？

Answer 1

Ruby插件可以为您提供帮助。

以下是配置：

input {
    stdin{
    }
}

filter {
    ruby {
        code => '
            msg = event["message"]
            msgIndex = msg.index("date=")
            msgInsert = msg[msgIndex..-1]
            msgMap = msgInsert.scan(/(\w+)=("(.*?)"|([^ ]+))/).map { |(first, second)| [first, second] }
            for x in msgMap
                key = x[0]
                value = x[1]
                event[key] = value
            end
        '
    }
}

output {
    stdout{
        codec => rubydebug
    }
}

首先，通过索引获取所有key = value对的起始值date=
然后将所有键值映射到字符串数组。
使用For循环插入所有值。

我已经尝试了你的日志，我可以用值创建所有对应的字段。希望这可以帮到你

Answer 2

避免grokparsefailure的简单答案是提供与您的输入相匹配的有效模式。也就是说，您的问题似乎暗示字段并不总是按此顺序指定。举个例子，你应该使用＆＃34; kv＆＃34;过滤以将这些键/值对拆分为字段。

正则表达式配置为logstash中的动态列

2 个答案: