我正在尝试解析一些本地日志文件，我正在我的Windows机器上运行ELK堆栈。这是我试图解析的日志的一个例子。
2015-12-10 13:50:25,487 [http-nio-8080-exec-26] INFO a.b.c.v1.myTestClass [abcde-1234-12345-b425-12ad]- This Message is OK
2015-12-10 13:50:26,487 [http-nio-8080-exec-26] INFO a.b.c.v1.myTestClass [abcde-1234-12345-b425-12ad]- Journe
y road update: <rows>
<row adi="D" date="2015-12-10" garage="TOP">
<codeNum order="1">TP</codeNum>
<number order="1">1001</number>
<journeystatus code="RT">OnRoute</journeystatus>
</row>
</rows>
第一条消息在过滤器中正常工作，但第二条消息在标签部分中被_grokparsefailure拆分为多条消息。
Logstash配置文件
input {
file {
path => "C:/data/sampleLogs/temp.log"
type => "testlog"
start_position => "beginning"
}
}
filter {
grok {
# Parse timestamp data. We need the "(?m)" so that grok (Oniguruma internally) correctly parses multi-line events
match => [ "message", [
"(?m)%{TIMESTAMP_ISO8601:logTimestamp}[ ;]\[%{DATA:threadId}\][ ;]%{LOGLEVEL:logLevel}[ ;]+%{JAVACLASS:JavaClass}[ ;]%{SYSLOG5424SD:TransactionID}[ ;]*%{GREEDYDATA:LogMessage}",
"(?m)%{TIMESTAMP_ISO8601:logTimestamp}[ ;]\[%{DATA:threadId}\][ ;]%{LOGLEVEL:logLevel}[ ;]+%{JAVAFILE:JavaClass}[ ;]%{SYSLOG5424SD:TransactionID}[ ;]*%{GREEDYDATA:LogMessage}"
]
]
}
# The timestamp may have commas instead of dots. Convert so as to store everything in the same way
mutate {
gsub => [
# replace all commas with dots
"logTimestamp", ",", "."
]
}
mutate {
gsub => [
# make the logTimestamp sortable. With a space, it is not! This does not work that well, in the end
# but somehow apparently makes things easier for the date filter
"logTimestamp", " ", ";"
]
}
date {
locale => "en"
timezone => "UTC"
match => [ "logTimestamp", "YYYY-MM-dd;HH:mm:ss.SSS" ]
target => "@timestamp"
}
mutate {
add_field => { "debug-timestamp" => "timestampMatched"}
}
}
output {
stdout {
codec => rubydebug
}
}
当我跑步时
bin\logstash agent -f \ELK-Stack\logstash\conf\01_input.conf
在CMD提示符中返回的内容如下
io/console not supported; tty will not be manipulated
Default settings used: Filter workers: 4
Logstash startup completed
{
"message" => " <row adi=\"D\" date=\"2015-12-10\" garage=\"TOP\"
>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.268Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => " <codeNum order=\"1\">TP</codeNum>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.268Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => " <number order=\"1\">1001</number>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.268Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => " <journeystatus code=\"RT\">OnRoute</journeys
tatus>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.278Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => " </row>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.278Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => "y road update: <rows>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.268Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => "2015-12-10 13:50:25,487 [http-nio-8080-exec-26] INFO
a.b.c.v1.myTestClass [abcde-1234-12345-b425-12ad]- Journe\r",
"@version" => "1",
"@timestamp" => "2015-12-10T13:50:25.487Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"logTimestamp" => "2015-12-10;13:50:25.487",
"threadId" => "http-nio-8080-exec-26",
"logLevel" => "INFO",
"JavaClass" => "a.b.c.v1.myTestClass",
"TransactionID" => "[abcde-1234-12345-b425-12ad]",
"LogMessage" => "- Journe\r",
"debug-timestamp" => "timestampMatched"
}
{
"message" => "</rows>2015-12-10 13:50:25,487 [http-nio-8080-exec-26]
INFO a.b.c.v1.myTestClass [abcde-1234-12345-b425-12ad]- This Message is OK\r",
"@version" => "1",
"@timestamp" => "2015-12-10T13:50:25.487Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"logTimestamp" => "2015-12-10;13:50:25.487",
"threadId" => "http-nio-8080-exec-26",
"logLevel" => "INFO",
"JavaClass" => "a.b.c.v1.myTestClass",
"TransactionID" => "[abcde-1234-12345-b425-12ad]",
"LogMessage" => "- This Message is OK\r",
"debug-timestamp" => "timestampMatched"
}
我确实在我的过滤器的顶部添加了mulitline，但它没有工作只是给了以下错误，就在grok之后。
multiline {
pattern => "^201*-**-**- **:**:"
what => "previous"
negate=> true
}
但这并没有帮助我不断给我一个错误消息，如
Error: Cannot use more than 1 filter worker because the following plugins don't
work with more than one worker: multiline
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.
所以我尝试按照建议运行--configtest，然后会出现一条新的错误消息
Error: Cannot use more than 1 filter worker because the following plugins don't
work with more than one worker: multiline
任何人都可以帮我解决这个问题，并让logstash处理多行。
非常感谢您的帮助
更新
@Alain Collins建议将编解码器与multiline一起使用，这是我的配置输入的样子。
input {
file {
path => "C:/data/sampleLogs/mulline.log"
codec => multiline {
# Grok pattern names are valid! :)
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
type => "testlog"
start_position => "beginning"
}
}
G
答案 0 :(得分：2)
您找到了正确的解决方案 - 多线。这些线条需要连接成一个事件。
如您所发现的，多行过滤器不是线程安全的，因此您只能在该logstash中运行一个工作线程。
有multiline codec可能适合您。它将行组合为输入{}阶段的一部分，并将一个事件传递给过滤器{}阶段。
请注意，您可以使用带有多行的logstash模式，因此“^％{YEAR}”会比“^ 201”更好。
最后，请关注filebeat，它是logstash-forwarder的替代品。他们说计划了客户端多线支持，因此消息将作为一个事件从客户端发送，而不必由logstash重新组装。