Question

这是我的推特输入推文

"_source": {
"created_at": "Wed Aug 10 06:42:48 +0000 2016",
"id": 763264318242783200,
"timestamp_ms": "1470811368891",
"@version": "1",
"@timestamp": "2016-08-10T06:42:48.000Z"
}

和我的logstash配置文件，其中包含twitter输入plugin过滤器和输出

input {
twitter {
consumer_key => "lvvoeonCRBOHsLAoTPbion9sK"
consumer_secret => "GNHOFzErJhuo0bNq38JUs7xea2BOktMiLa7tunoGwP0oFKCHrY"
oauth_token => "704578110616936448-gfeSklNrITu7fHIZgjw3nwoZ1S0l0Jl"
oauth_token_secret => "IHiyRJRN09jjdUTGrnesALw4DRle35WyX7pdnI3CtEnJ5"
keywords => [ "afghanistan", "TOLOnews", "kabul", "police"]
full_tweet => true
}
}
filter {
    date {
      match => ["timestamp" , "MMM d YYY HH:mm:ss", "ISO8601"]
  }
 }
output {
   stdout { codec => dots }
    elasticsearch {
        hosts => "10.20.1.123"
        index => "twitter_news"
        document_type => "tweets"
    }
}

我想今天刚刚收到新推文的日期是2016-11-16，那么我只想获得@timestamp= 2016-11-16而不是@timestamp= 2016-11-15或过去几天推文的推文，但是这个配置我也接过推文，任何人都帮我这个怎么做？

Answer 1

这里的想法是在logstash配置中使用ruby代码。我建议使用timestamp_ms来比较日期。

首先需要将timestamp_ms转换为整数
使用ruby

比较时间戳这是一个例子：

mutate {
    convert => {
        "timestamp_ms" => "integer"
    }
}
ruby {
    code => "
        t = Time.now
        today_ymd = t.strftime('%Y%m%d')
        today_timestamp_ms = DateTime.parse(today_ymd).to_time.to_i*1000
        event['@metadata']['today_timestamp_ms'] = today_timestamp_ms
    "
}


if [timestamp_ms] < [@metadata][today_timestamp_ms] {
## past days events
    mutate {
        add_field => { "test" => "past days events"  }
    }
} else {
# today events
    mutate {
        add_field => { "test" => "today events"  }
    }

}

如何根据日期提交过滤logstash的输入数据？

1 个答案: