Question

我有一个

形式的json

[
    {
        "foo":"bar"
    }
]

我正在尝试使用logstash中的json过滤器对其进行过滤。但它似乎没有用。我发现我无法使用logstash中的json过滤器解析列表json。有人可以告诉我有关解决方法吗？

更新

我的日志

IP - - 0.000 0.000 [24/May/2015:06:51:13 +0000] *"POST /c.gif HTTP/1.1"* 200 4 * user_id=UserID&package_name=SomePackageName&model=Titanium+S202&country_code=in&android_id=AndroidID&eT=1432450271859&eTz=GMT%2B05%3A30&events=%5B%7B%22eV%22%3A%22com.olx.southasia%22%2C%22eC%22%3A%22appUpdate%22%2C%22eA%22%3A%22app_activated%22%2C%22eTz%22%3A%22GMT%2B05%3A30%22%2C%22eT%22%3A%221432386324909%22%2C%22eL%22%3A%22packageName%22%7D%5D * "-" "-" "-"

以上日志的URL解码版本是

IP - - 0.000 0.000 [24/May/2015:06:51:13  0000] *"POST /c.gif HTTP/1.1"* 200 4 * user_id=UserID&package_name=SomePackageName&model=Titanium S202&country_code=in&android_id=AndroidID&eT=1432450271859&eTz=GMT+05:30&events=[{"eV":"com.olx.southasia","eC":"appUpdate","eA":"app_activated","eTz":"GMT+05:30","eT":"1432386324909","eL":"packageName"}] * "-" "-" "-"

请在下面的配置文件中找到以上日志..

过滤{

urldecode{
    field => "message"
}
 grok {
  match => ["message",'%{IP:clientip}%{GREEDYDATA} \[%{GREEDYDATA:timestamp}\] \*"%{WORD:method}%{GREEDYDATA}']
}

kv {
    field_split => "&? "
}
json{
    source=> "events"
}
geoip {
    source => "clientip"
}

}

我需要解析事件，即events=[{"eV":"com.olx.southasia","eC":"appUpdate","eA":"app_activated","eTz":"GMT+05:30","eT":"1432386324909","eL":"packageName"}]

Answer 1

我假设你的json在一个文件中。你是对的，你不能直接使用json过滤器。您必须使用多行编解码器，然后使用json过滤器。

以下配置适用于您的输入。但是，您可能需要更改它才能正确分隔事件。这取决于您的需求和文件的json格式。

Logstash config：

input     {   
    file     {
        codec => multiline
        {
            pattern => "^\]" # Change to separate events
            negate => true
            what => previous               
        }
        path => ["/absolute/path/to/your/json/file"]
        start_position => "beginning"
        sincedb_path => "/dev/null" # This is just for testing
    }
}

filter     {
    mutate   {
            gsub => [ "message","\[",""]
            gsub => [ "message","\n",""]
        }
    json { source => message }
}

更新

更新后，我猜我发现了问题。显然，由于方括号，你得到了一个 jsonparsefailure 。作为解决方法，您可以手动删除它们。在kv之后和json过滤器之前添加以下mutate过滤器：

mutate  {
    gsub => [ "events","\]",""]
    gsub => [ "events","\[",""]
}

更新2

好吧，假设您的输入如下：

[{"foo":"bar"},{"foo":"bar1"}]

以下是4个选项：

选项a）丑陋的gsub

一个丑陋的解决方法是另一个gsub：

gsub => [ "event","\},\{",","]

但这会消除内在关系，所以我猜你不想这样做。

选项b）拆分

更好的方法可能是使用拆分过滤器：

split {
    field => "event"
    terminator => ","
}
mutate  {
    gsub => [ "event","\]",""]
    gsub => [ "event","\[",""]
   }
json{
    source=> "event"
}

这将生成多个事件。（首先是foo = bar，第二个是foo1 = bar1。）

选项c）改变分裂

您可能希望在一个logstash事件中包含所有值。你可以使用 mutate =＆gt;拆分过滤器以生成数组并在条目存在时解析json。不幸的是，你必须为每个条目设置一个条件，因为logstash不支持其配置中的循环。

mutate  {
    gsub => [ "event","\]",""]
    gsub => [ "event","\[",""]
    split => [ "event", "," ]
   }

json{
    source=> "event[0]"
    target => "result[0]"
}

if 'event[1]' {
    json{
        source=> "event[1]"
        target => "result[1]"
    }
    if 'event[2]' {
        json{
            source=> "event[2]"
            target => "result[2]"
        }
    }
    # You would have to specify more conditionals if you expect even more dictionaries
}

选项d）Ruby

根据你的评论，我试图找到一种红宝石的方式。以下工作（在您的kv过滤器之后）：

mutate  {
    gsub => [ "event","\]",""]
    gsub => [ "event","\[",""]
}

ruby  {
    init => "require 'json'"
    code => "
        e = event['event'].split(',')
        ary = Array.new
        e.each do |x|
            hash = JSON.parse(x)
            hash.each do |key, value|
                ary.push( { key =>  value } )
            end
        end
        event['result'] = ary
    "
}

选项e）Ruby

在kv过滤器之后使用此方法（不设置mutate过滤器）：

ruby  {
    init => "require 'json'"
    code => "
            event['result'] = JSON.parse(event['event'])
    "
}

它将解析event=[{"name":"Alex","address":"NewYork"},{"name":"David","address":"NewJersey"}]

等事件

成：

"result" => [
    [0] {
           "name" => "Alex",
        "address" => "NewYork"
    },
    [1] {
           "name" => "David",
        "address" => "NewJersey"
    }

由于kv过滤器的行为不支持空格。我希望你没有真正的投入，是吗？

在logstash

1 个答案:

更新

更新2