Question

我正在使用Logstash，而且我遇到了一个很简单的配置工作的麻烦。

input {
  file {
    path => "C:/path/test-data/*.log"
    start_position => beginning
    type => "usage_data"
  }
}

filter {
  if [type] == "usage_data" {
    grok {
      match => { "message" => "^\s*%{NUMBER:lineNumber}\s+%{TIMESTAMP_ISO8601:date},(?<value1>[A-Za-z0-9+/]+),(?<value2>[A-Za-z0-9+/]+),(?<value3>[A-Za-z0-9+/]+),(?<value4>[^,]+),(?<value5>[^\r]*)" }
    }
  }

  if "_grokparsefailure" not in [tags] {
    drop { }
  }
}

output {
  stdout { codec => rubydebug }
}

我这样称呼Logstash：

SET LS_MAX_MEM=2g

DEL "%USERPROFILE%\.sincedb_*" 2> NUL
"C:\Program Files (x86)\logstash-1.4.1\bin\logstash.bat" agent -p "C:\path\\." -w 1 -f "logstash.conf"

输出：

←[33mUsing milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.1/plugin-milestones {:level=>:w
arn}←[0m
{
       "message" => ",",
      "@version" => "1",
    "@timestamp" => "2014-11-20T09:16:08.591Z",
          "type" => "usage_data",
          "host" => "my-machine",
          "path" => "C:/path/test-data/monitor_20141116223000.log",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

如果我只解析C:\path\test-data\monitor_20141116223000.log，则会读取所有行，并且没有grokparsefailure。如果我删除C:\path\test-data\monitor_20141116223000.log，则会在另一个日志文件中弹出相同的grokparsefailure：

{
       "message" => "atches in another context\r",
      "@version" => "1",
    "@timestamp" => "2014-11-20T09:14:04.779Z",
          "type" => "usage_data",
          "host" => "my-machine",
          "path" => "C:/path/test-data/monitor_20140829235900.log",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

特别是最后一个输出证明Logstash没有读取整行或尝试解释没有的换行符。它总是在相同位置的同一条线上断开。

也许我应该添加日志文件包含\n作为行分隔符，并且我在Windows上运行Logstash。但是，我没有收到很多错误，只有那个错误。那里有很多行。当我删除if "_grokparsefailure" ...时，它们都显示正确。

我认为缓冲有一些问题，但我不知道如何使这项工作。有什么想法吗？

Answer 1

解决方法：

# diff -Nur /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb.orig /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb
--- /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb.orig       2015-02-25 10:46:06.916321816 +0700
+++ /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb    2015-02-12 18:39:34.943833909 +0700
@@ -86,7 +86,9 @@
           _read_file(path, &block)
           @files[path].close
           @files.delete(path)
-          @statcache.delete(path)
+          #@statcache.delete(path)
+          inode = @statcache.delete(path)
+          @sincedb[inode] = 0
         else
           @logger.warn("unknown event type #{event} for #{path}")
         end

Logstash不会使用文件输入读取整行

1 个答案: