Question

我在尝试找出以下消息类型的grok表达式时遇到了很多麻烦（来自Sophos UTM）

Apr 28 16:57:49 utm-vap-xx.domain.local 2018：04：28-17：02：05 s-utm-01 httpproxy [52816]：id =“0001”severity =“info” sys =“SecureWeb”sub =“http”name =“http access”action =“pass”method =“POST”srcip =“10.11.110.5”dstip =“216.163.176.36”user =“”group =“”ad_domain = “”statuscode =“200”cached =“0”profile =“REF_DefaultHTTPProfile（默认Web过滤器配置文件）”filteraction =“REF_DefaultHTTPCFFAction（默认内容过滤器操作）”size =“15”request =“0xdae2cc00”url =“http：/ /iprep3.t.ctmail.com/SpamResolverNG/SpamResolverNG.dll?DoNewRequest“referer =”“error =”“authtime =”0“dnstime =”905“cattime =”143“avscantime =”2275“fullreqtime =”238344“ device =“0”auth =“0”ua =“Mozilla / 4.0（兼容; Win32; Commtouch Http客户端（curl））”exceptions =“”category =“178”reputation =“neutral”categoryname =“Internet Services”国家=“美国”content-type =“text / html”sandbox =“ - ”

当我想跳过字段或值对包含空字符串时，会出现问题。例如，在Logstash中添加类似

的内容

  beforeEach(async(() => {
    TestBed.configureTestingModule({
      imports: [
        AngularFireModule.initializeApp(environment.firebase)
      ],
      declarations: [
        BlogPostComponent,
        HeaderComponent,
        FooterComponent
      ]
    }).compileComponents();

    fixture = TestBed.createComponent(BlogPostComponent);
    component = fixture.componentInstance;
    element = fixture.nativeElement;
    de = fixture.debugElement;

    fixture.detectChanges();
  }));

导致问题，而这些问题在联机grok builder

中有效

目标是得到像

这样的东西

()?(srcip={%"IP:SourceIP"})

我也打算在Logstash中使用geo-tags，我已经与其他来源合作了。

期待获得一些有价值的帮助。感谢

Answer 1

有人撰写了Grok Sophos UTM 9.x Pattern for logstash，

极速增强的Sophos UTM ulogd和pluto日志的Grok模式各种格式的消息，并将消息体与头

 grok {
  pattern => ['(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGHOST:logsource}) (?:%{YEAR}): (?:%{MONTHNUM}):(?:%{MONTHDAY})-(?:%{HOUR}):(?:%{MINUTE}):(?:%{SECOND}) (?:%{SYSLOGHOST}) (?:%{SYSLOGPROG}): (?<messagebody>(?:id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" tcpflags=\"%{DATA:utm_ulogd_tcpflags}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" reason=\"%{DATA:utm_ips_reason}\" group=\"%{INT:utm_ips_group}\" srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" sid=\"%{INT:utm_ips_sid}\" class=\"%{DATA:utm_ips_class}\" priority=\"%{INT:utm_ips_priority}\" generator=\"%{INT:utm_ips_generator}\" msgid=\"%{INT:utm_ips_msgid}\"|\"%{DATA:utm_pluto_vpnname}\"\[%{INT}\] %{IP:utm_pluto_vpnremoteip} #%{INT}: %{GREEDYDATA:utm_pluto_message}|%{GREEDYDATA}))']
  type => "sophosutm"
 }

这适用于你的日志（我测试过它）

但是，如果您对作为字段分隔的整个数据不感兴趣，并且只对问题中提到的特定数据感兴趣，那么您可以将不必要的数据指定为GREEDYDATA，并仅提取所需的字段，如下所示，

sub=%{QUOTEDSTRING:protocol}\s*.*\s*srcip=\"%{IP:SourceIP}\"\s*dstip=\"%{IP:DestinationIP}\"

上述grok模式将提取 Sub，SourceIP和目标IP ，并生成以下输出，

 {
  "protocol": [
    [
      ""http""
    ]
  ],
  "SourceIP": [
    [
      "10.11.110.5"
    ]
  ],
  "IPV6": [
    [
      null,
      null
    ]
  ],
  "IPV4": [
    [
      "10.11.110.5",
      "216.163.176.36"
    ]
  ],
  "DestinationIP": [
    [
      "216.163.176.36"
    ]
  ]
}

可以使用相同的模式过滤更多数据。

Answer 2

这就是我用来获取modsecurity消息的原因。我从来没有回过头来将表达式简化为一个，所以它们都被调出了，但是这对于您reverseproxy.log或任何不是直接KV的东西都应该起作用。

if "ModSecurity:" in [message] {
    grok {
    break_on_match => false
        match => [
        "message", ' \[hostname %{QUOTEDSTRING:Hostname}\] \[client %{IPORHOST:Source_IP}\]'
        ] #end match
    } #end grok
     grok {
    break_on_match => false
        match => [
        "message", ' \[client %{IPORHOST:Source_IP}\]'
        ] #end match
    } #end grok
    grok {
    break_on_match => false
        match => [
        "message", ' \[severity %{QUOTEDSTRING:Rule_Severity}\]'
        ] #end match
    } #end grok
    grok {
    break_on_match => false
        match => [
        "message", ' \[id %{QUOTEDSTRING:Rule_ID}\]'
        ] #end match
    } #end grok
    grok {
    break_on_match => false
        match => [
        "message", ' \[uri %{QUOTEDSTRING:Target_URI}\]'
        ] #end match
         add_field => [ "Logsource" , "Reverse Proxy (Modsecurity)" ]
    } #end grok
    grok {
    break_on_match => false
        match => [
        "message", '\[msg %{QUOTEDSTRING:MSG}\]'
        ] #end match
        add_field => [ "received_at", "%{@timestamp}" ]
    } #end grok
    date {
        match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
    } #end date
    mutate {
    } # end mutate
  } #end "if ModSecurity:"

Sophos syslog的grok表达式

2 个答案: