Question

我正在尝试过滤存储在文件夹中的3个错误文件，但无法将其过滤掉。 Kibana将所有三个文件合并为一个，并且不过滤任何数据。我想用文件信号中的不同标记名（文件名）正确显示错误文件。任何帮助将不胜感激

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/mylog/*.log

output.logstash:
  hosts: ["logstash:5044"]

processors:
  - add_docker_metadata:
      host: "unix:///host_docker/docker.sock"
    add_tags:
      tags: ["accesslog"]

logstash.conf

input {
  beats {
    port => 5044
  }
}
filter {
  grok {
    match => { "message" => ["%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:log_level}\] \<%{DATA:field_misc}\> %{GREEDYDATA:message}"] }
  }
  json {
    source => "message"
  }
  date{
    match => ["timestamp", "UNIX_MS"]
    target => "@timestamp"
  }
  ruby {
    code => "event.set('indexDay', event.get('[@timestamp]').time.localtime('+09:00').strftime('%Y%m%d'))"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    template => "/usr/share/logstash/templates/logstash.template.json"
    template_name => "logstash"
    template_overwrite => true
    index => "logstash-%{indexDay}"
    codec => json
  }
  stdout {
    codec => rubydebug
  }
}

访问日志：

DEBUG: 2020/09/18 22:40:28 adserver.go:504: RabbitMQ received a message:  {"order_id": "AdServer.OrderID:3541029", "tag": "<?xml version=\"1.0\" encoding=\"utf-8\"?><VAST version=\"2.0\"><Ad id=\"3541029\"><InLine><AdSystem version=\"1.0\">ExtendTV</AdSystem><AdTitle>dg_creative_2020_9_18_22_39__7386_2089518</AdTitle><Impression id=\"0\"><![CDATA[https://adserver-east.extend.tv/video/?event=imp&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Impression><Error><![CDATA[https://adserver-east.extend.tv/video/?event=err&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Error><Impression id=\"truoptik\"><![CDATA[https://dmp.truoptik.com/e4dded9c70efc910/sync.gif?]]></Impression><Impression id=\"liveramp\"><![CDATA[https://sync.extend.tv/r.gif?exchange=liveramp&b=[b]]]></Impression><Creatives><Creative><Linear><Duration>00:00:10</Duration><VideoClicks><ClickThrough><![CDATA[http://www.susaludclinic.com/]]></ClickThrough><ClickTracking><![CDATA[https://adserver-east.extend.tv/video/?event=clk&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></ClickTracking></VideoClicks><TrackingEvents><Tracking event=\"thirdQuartile\"><![CDATA[https://adserver-east.extend.tv/video/?event=3rd&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"pause\"><![CDATA[https://adserver-east.extend.tv/video/?event=pse&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"complete\"><![CDATA[https://adserver-east.extend.tv/video/?event=view&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"resume\"><![CDATA[https://adserver-east.extend.tv/video/?event=res&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"unmute\"><![CDATA[https://adserver-east.extend.tv/video/?event=unmut&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"fullscreen\"><![CDATA[https://adserver-east.extend.tv/video/?event=fscr&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"mute\"><![CDATA[https://adserver-east.extend.tv/video/?event=mut&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"midpoint\"><![CDATA[https://adserver-east.extend.tv/video/?event=mid&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"firstQuartile\"><![CDATA[https://adserver-east.extend.tv/video/?event=1st&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking><Tracking event=\"close\"><![CDATA[https://adserver-east.extend.tv/video/?event=cls&entity=25&order=3541029&version=2.0&b=[b]&p=[p]]]></Tracking></TrackingEvents><MediaFiles><MediaFile bitrate=\"720\" delivery=\"progressive\" height=\"360\" type=\"video/mp4\" width=\"640\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_lo.mp4]]></MediaFile><MediaFile bitrate=\"720\" delivery=\"progressive\" height=\"360\" type=\"video/webm\" width=\"640\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_lo.webm]]></MediaFile><MediaFile bitrate=\"720\" delivery=\"progressive\" height=\"360\" type=\"video/x-flv\" width=\"640\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_lo.flv]]></MediaFile><MediaFile bitrate=\"1200\" delivery=\"progressive\" height=\"480\" type=\"video/mp4\" width=\"854\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_me.mp4]]></MediaFile><MediaFile bitrate=\"1200\" delivery=\"progressive\" height=\"480\" type=\"video/webm\" width=\"854\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_me.webm]]></MediaFile><MediaFile bitrate=\"1200\" delivery=\"progressive\" height=\"480\" type=\"video/x-flv\" width=\"854\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_me.flv]]></MediaFile><MediaFile bitrate=\"2400\" delivery=\"progressive\" height=\"720\" type=\"video/mp4\" width=\"1280\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_hi.mp4]]></MediaFile><MediaFile bitrate=\"2400\" delivery=\"progressive\" height=\"720\" type=\"video/webm\" width=\"1280\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_hi.webm]]></MediaFile><MediaFile bitrate=\"2400\" delivery=\"progressive\" height=\"720\" type=\"video/x-flv\" width=\"1280\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_hi.flv]]></MediaFile><MediaFile bitrate=\"4000\" delivery=\"progressive\" height=\"1080\" type=\"video/mp4\" width=\"1920\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_hd.mp4]]></MediaFile><MediaFile bitrate=\"4000\" delivery=\"progressive\" height=\"1080\" type=\"video/webm\" width=\"1920\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_hd.webm]]></MediaFile><MediaFile bitrate=\"4000\" delivery=\"progressive\" height=\"1080\" type=\"video/x-flv\" width=\"1920\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_hd.flv]]></MediaFile><MediaFile bitrate=\"15000\" delivery=\"progressive\" height=\"1080\" type=\"video/mp4\" width=\"1920\"><![CDATA[https://s4w9z8k2.ssl.hwcdn.net/videos/dg_creative_2020_9_18_22_39__7386_2089518_mp4/dg_creative_2020_9_18_22_39__7386_2089518_hd_15000.mp4]]></MediaFile></MediaFiles></Linear></Creative></Creatives></InLine></Ad></VAST>", "clickthru": "http://www.susaludclinic.com/"}
DEBUG: 2020/09/18 22:40:28 adserver.go:511: In readFromMQ, received Order: 3541029
DEBUG: 2020/09/18 22:58:14 adserver.go:1673: /
DEBUG: 2020/09/18 22:58:14 adserver.go:1692: RequestURI: /
DEBUG: 2020/09/18 23:06:16 adserver.go:1673: /
DEBUG: 2020/09/18 23:06:16 adserver.go:1692: RequestURI: /
DEBUG: 2020/09/18 23:23:21 adserver.go:1673: /.env
DEBUG: 2020/09/18 23:23:21 adserver.go:1692: RequestURI: /.env
DEBUG: 2020/09/18 23:23:22 adserver.go:1673: /
DEBUG: 2020/09/18 23:23:22 adserver.go:1692: RequestURI: /

错误日志：

ERROR: 2020/09/18 22:07:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48276->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 22:17:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48258->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 22:27:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48260->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 22:37:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48262->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 22:47:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48264->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 22:57:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48266->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 23:07:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48268->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 23:17:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48270->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 23:27:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48272->10.0.0.213:6379: use of closed network connection
ERROR: 2020/09/18 23:37:16 adserver.go:366: hgetall on redis_entitylist_key ERROR: write tcp 172.19.0.2:48274->10.0.0.213:6379: use of closed network connection

事件日志

2020/07/15 19:25:12 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN 1
2020/07/15 19:25:12     1594841112  2.0 172.21.0.1:39436    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 19:34:44 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count map[accname:SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN accnamenoex:Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN adGroupId:Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN adType:4 app_id:SpotX_Roku OS_46041 aud: bidprice:22.00 brand:Roku bundleid:46041 cAdType:0 campaignId:SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video carrier: cat: city:Goodland creativeId:2079953 dealid:30907.885f3.46db device:7 deviceid:e0d2403fd48b73ed7245eff2021b3d9d dma:678 exchId: extendId: geocriteria:0 height:1080 idl: imp_count:3 ip:68.232.59.208 latitude: longitude: model:Digital Video player platform: playersize: pubId:198919 skip: slotvis: state:Kansas timestamp:1594839703 ua:Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) url:https://channelstore.roku.com/details/46041 urlhost:app:Sling TV view_count:1 width:1920 winprice:0.0006818 zip:67735]
2020/07/15 19:34:44 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN 1
2020/07/15 19:34:44     1594841684  2.0 172.21.0.1:39600    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 20:11:17     1594843877  2.0 172.21.0.1:40362    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 20:56:49     1594846609  2.0 172.21.0.1:41216    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208

Answer 1

可以通过添加字段在Filebeat级别标记不同类型的日志文件。

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /PATH/TO/ACCESS.log
  fields:
    category: access_log
- type: log
  enabled: true
  paths:
    - /PATH/TO/ERROR.log
  fields:
    category: error_log

上面的配置添加了一个新的字段，用于由日志的各个类别处理的文档。

您可以在Kibana中输入（例如）字段名称“ fields.category：error_log”以过滤错误日志

如何使用logtash和filebeat过滤日志文件

1 个答案: