我遇到的问题是我尝试使用IIS日志和Elasticsearch解决。发生的事情是IIS日志中的用户名中包含反斜杠（）而不是正斜杠（/）。当Elasticsearch返回名称时，它不再具有\我希望在输入时可以转义的\。因此，当在Elasticsearch或Kibana中查看结果时，用户名没有\，并且那里的斜杠被视为正则表达式。一个例子是用户名abcd \ bob将作为abcdob返回。
我也相信这个问题是我在IIS中为每个条目添加_grokparsefailure标记的原因。
有什么建议吗？
获取数据的我的NXLOG文件：
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension json>
Module xm_json
</Extension>
#<Extension w3c>
#map iis log fields to Field Types
# Module xm_csv
# Fields $date, $time, $website, $serverip, $method, $url, $querystring, $port, $username, $clientip, $version, $useragent, $referer, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
# FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
#
# #Fields $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
# #FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
# Delimiter ' '
# QuoteChar '"'
# EscapeControl FALSE
# UndefValue -
#</Extension>
<Extension w3c>
#map iis log fields to Field Types
Module xm_csv
Fields $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer
Delimiter ' '
</Extension>
# Nxlog internal logs
<Input internal>
Module im_internal
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
# Uncomment im_mseventlog for Windows XP/2000/2003
# Module im_mseventlog
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
<Input iis-logs>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
ReadFromLast TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
to_json (); \
}
</Input>
<Output out>
Module om_tcp
Host logs.{domain removed}.com
Port 3515
</Output>
<Output iis-out>
Module om_tcp
Host logs.{domain removed}.com
Port 3516
</Output>
<Route 1>
Path internal, eventlog => out
</Route>
<Route 2>
Path iis-logs => iis-out
</Route>
我的Logstash.conf文件：
input {
tcp {
port => 5000
type => "syslog"
}
tcp {
type => "eventlog"
port => 3515
codec => json_lines
}
tcp {
type => "iislog"
port => 3516
codec => json_lines
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{DATA:syslog_program}\[%{NUMBER:syslog_pid}\]\: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
# add_field => [ "received_from", "%{@source_host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
}
if "_grokparsefailure" not in [tags] {
mutate {
replace => [ "@message", "%{syslog_message}" ]
}
}
mutate {
remove => [ "syslog_message", "syslog_timestamp" ]
}
kv {
source => "@message"
}
}
if [type] == "eventlog" {
# Incoming Windows Event logs from nxlog
# The EventReceivedTime field must contain only digits, or it is an invalid message
# if [EventReceivedTime] !~ /\d+/ { drop { } }
# grep {
# match => [ "EventReceivedTime", "\d+" ]
# }
mutate {
# Lowercase some values that are always in uppercase
lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
}
mutate {
# Set source to what the message says
rename => [ "Hostname", "@source_host" ]
}
date {
# Convert timestamp from integer in UTC
match => [ "EventReceivedTime", "UNIX" ]
}
mutate {
# Rename some fields into something more useful
rename => [ "Message", "@message" ]
rename => [ "Severity", "eventlog_severity" ]
rename => [ "SeverityValue", "eventlog_severity_code" ]
rename => [ "Channel", "eventlog_channel" ]
rename => [ "SourceName", "eventlog_program" ]
rename => [ "SourceModuleName", "nxlog_input" ]
rename => [ "Category", "eventlog_category" ]
rename => [ "EventID", "eventlog_id" ]
rename => [ "RecordNumber", "eventlog_record_number" ]
rename => [ "ProcessID", "eventlog_pid" ]
}
mutate {
# Remove redundant fields
remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
}
if [eventlog_id] == 4624 {
mutate {
add_tag => [ "ad-logon-success" ]
}
}
if [eventlog_id] == 4634 {
mutate {
add_tag => [ "ad-logoff-success" ]
}
}
if [eventlog_id] == 4771 or [eventlog_id] == 4625 or [eventlog_id] == 4769 {
mutate {
add_tag => [ "ad-logon-failure" ]
}
}
if [eventlog_id] == 4723 {
mutate {
add_tag => [ "ad-password-change" ]
}
}
if [eventlog_id] == 4724 {
mutate {
add_tag => [ "ad-password-reset" ]
}
}
if "ad-logon-success" in [tags] {
metrics {
add_tag => [ "drop", "metric", "ad-logon-success" ]
meter => "ad-logon-success-metric"
}
}
if "ad-logon-failure" in [tags] {
metrics {
add_tag => [ "drop", "metric", "ad-logon-failure" ]
meter => "ad-logon-failure-metric"
}
}
}
if [type] == "iislog"
{
grok {
# match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{IP:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"]
match => ["message", "%{DATESTAMP:log_timestamp} %{WORD:sitename} %{HOSTNAME:computername} %{IP:hostip} %{URIPROTO:method} %{URIPATH:request} (?:%{NOTSPACE:queryparam}|-) %{NUMBER:port} (?:%{NOTSPACE:username}|-) %{IP:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:user-agent} (?:%{NOTSPACE:cookie}|-) (?:%{NOTSPACE:referer}|-) (?:%{HOSTNAME:host}|-) %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"]
}
useragent {
source => "useragent"
}
#geoip {
# source => "clientip"
#}
}
metrics {
meter => "events"
add_tag => [ "drop", "metric", "events-metric" ]
}
}
output {
if "drop" not in [tags] {
elasticsearch {
host => "127.0.0.1"
cluster => "logs"
}
# stdout { codec => rubydebug }
}
}
示例IIS日志条目：
<section id="intro">
<div class="wrap">
<div class="content">
<h1 id="welcome"> Your number one choice for music <br> in the south coast </h1>
</div>
</div>
</section>
更新 我添加了第二个nxlog iis输入/输出，而不是将数据输出到我的logstash服务器，我输出到一个平面文件。
<Output iis2-out>
Module om_file
File 'C:\logs\logtest.txt'
</Output>
我检查了输出，发现用户在点击logstash服务器之前已从用户名中删除了反斜杠。