Logstash 6.1.1不从Filebeat 6.1.1接收数据：ERR连接失败

时间：2018-01-11 09:45:09

标签： logstash logstash-configuration filebeat logstash-file

我正在尝试使用logstash与filebeat进行通信。它部分有效。我有两个节点。首先是filebeat-node，这是filebeat.yml：

filebeat.prospectors:
- type: log
enabled: true
paths:
- /home/centos/logs/*.log  
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
output.logstash:
hosts: ["10.206.81.239:5044"]

其次是logstash-node，logstash.yml如下所示：

path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/*.conf
path.logs: /var/log/logstash

Logstash管道first-pipelien.conf：

input {
    beats {
        port => "5044"
    }
    file{
        path => "/home/centos/logs/mylogs.log"
    }
}
filter {
    grok{
        match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
    }
}
output {
    elasticsearch {
        hosts => [ "10.206.81.246:9200", "10.206.81.236:9200", "10.206.81.243:9200" ]
    }   
     stdout { codec => rubydebug }
}

现在，当我按filebeat -e -d "publish"和/usr/share/logstash/bin/logstash -f /usr/share/logstash/first-pipeline.conf --config.reload.automatic开始文件节拍时，一切都差不多了。我通过echo "12.4.14.27 AAA /index.html 138 0.23">> /home/centos/logs/mylogs.log添加日志。使用我的日志在控制台json文件中打印Filebeat，然后打印：

ERR Failed to publish events caused by: write tcp 10.206.81.235:59922->10.206.81.239:5044: write: connection reset by peer
2018/01/11 08:56:17.774080 output.go:92: ERR Failed to publish events: write tcp 10.206.81.235:59922->10.206.81.239:5044: write: connection reset by peer

Logstash使用log打印相同的json并将其传递给elasticsearch（索引增加中的doc数）。该错误看起来像logstash重新启动连接，所以我希望需要配置更大的计时器来保持连接（尚未尝试）。但它在某种程度上起作用。

问题是，当我使用'systemctl start logstash'启动logstash时。 Logstash正在运行。当我向filebeat添加一些日志时，我收到了这个错误：

2018/01/11 08:39:32.355756 output.go:74: ERR Failed to connect: dial tcp 10.206.81.239:5044: getsockopt: connection refused
2018/01/11 08:39:34.357051 output.go:74: ERR Failed to connect: dial tcp 10.206.81.239:5044: getsockopt: connection refused
2018/01/11 08:39:38.358227 output.go:74: ERR Failed to connect: dial tcp 10.206.81.239:5044: getsockopt: connection refused
2018/01/11 08:39:46.359342 output.go:74: ERR Failed to connect: dial tcp 10.206.81.239:5044: getsockopt: connection refused

没有任何东西被发现。此外，在logstash节点上，/var/log/logstash/logstash-plain.log：

中存在错误 
[2018-01-11T09:51:28,235][ERROR][io.netty.util.concurrent.DefaultPromise.rejectedExecution] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:840) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:342) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:335) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:765) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:767) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:435) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:906) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:487) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:85) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:330) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:281) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:277) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:259) [netty-all-4.1.3.Final.jar:4.1.3.Final]
        at org.logstash.beats.Server.listen(Server.java:65) [logstash-input-beats-5.0.4.jar:?]
        at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438) [?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302) [?:?]
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [?:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:129) [?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_beats_minus_5_dot_0_dot_4_minus_java.lib.logstash.inputs.beats.invokeOther4:listen(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.4-java/lib/logstash/inputs/beats.rb:199) [?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_beats_minus_5_dot_0_dot_4_minus_java.lib.logstash.inputs.beats.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.4-java/lib/logstash/inputs/beats.rb:199) [?:?]
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103) [?:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163) [?:?]
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [?:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161) [?:?]
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314) [?:?]
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [?:?]
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83) [?:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179) [?:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165) [?:?]
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [?:?]
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338) [?:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163) [?:?]
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314) [?:?]
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [?:?]
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132) [?:?]
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148) [?:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73) [?:?]
        at org.jruby.runtime.Block.call(Block.java:124) [?:?]
        at org.jruby.RubyProc.call(RubyProc.java:289) [?:?]
        at org.jruby.RubyProc.call(RubyProc.java:246) [?:?]
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104) [?:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

ls -al /etc/logstash的输出：

drwxrwxr-- 2 logstash logstash   33 10. led 11.37 conf.d
-rwxrwxr-- 1 logstash logstash 1736  9. led 17.34 jvm.options
-rwxrwxr-- 1 logstash logstash 1334 17. pro 22.51 log4j2.properties
-rwxrwxr-- 1 logstash logstash 6454  9. led 16.06 logstash.yml
-rwxrwxr-- 1 logstash logstash 1659 17. pro 22.51 startup.options
drwxr-xr-x 2 root     root       47 11. led 09.52 ${sys:ls.logs}

我这样设置，因为我认为，logstash文件的权限存在问题。

版本：
logstash 6.1.1
filebeat版本6.1.1（amd64）

问题：

  1. 为什么它适用于logstash命令但不能作为守护进程？
  2. 如何删除ERR Failed to publish events错误？
  3. 如何删除ERR Failed to connect和Failed to submit a listener notification task. Event loop shut down?错误？
  4. 应该将哪些权限和所有者设置为/ etc / logstash目录中的所有文件？

0 个答案:

没有答案
相关问题
最新问题