Question

带有netflow模块的Logstash 6.2.4

Elasticsearch版本：6.2.4

Ubuntu 16.04 LTS

我遇到一个问题，其中logstash正在正确的端口上侦听，但似乎没有在收集 netflow数据并将其传递给elasticsearch。

我们网络中的路由器正在将其netflow数据发送到服务器A，而nfcap正在监听端口 9995，因此尝试使用服务器A上的netflow模块运行logstash会导致使用地址错误。因此，我正在使用iptables复制数据包，并将其转发到另一台服务器Server B，就像这样。

iptables -t mangle -A PREROUTING -p udp --dport 9995 -j TEE --gateway <Server B ip address>

通过tcpdump进行检查，我可以看到服务器B接收到的重复数据包以及服务器A的IP地址。输出在下面，但是出于安全原因，我已经编辑了IP地址。

tcpdump -i eno1 -n dst port 9995
12:49:49.130772 IP <Router 1 ip address>.10005 > <Server A ip address>.9995: UDP, length 1392
12:49:49.131067 IP <Router 1 ip address>.10005 > <Server A ip address>.9995: UDP, length 1392
12:49:49.133504 IP <Router 1 ip address>.10005 > <Server A ip address>.9995: UDP, length 1392
12:49:49.133527 IP <Router 1 ip address>.10005 > <Server A ip address>.9995: UDP, length 1392
12:49:49.133533 IP <Router 1 ip address>.10005 > <Server A ip address>.9995: UDP, length 1260
12:49:49.391871 IP <Router 2 ip address>.62500 > <Server A ip address>.9995: UDP, length 1452
12:49:49.391894 IP <Router 2 ip address>.62500 > <Server A ip address>.9995: UDP, length 1368

因此，我知道服务器B正在端口9995上接收数据包。使用netstat进行检查也显示了这一点。

netstat -an | grep 9995
udp        0      0 0.0.0.0:9995            0.0.0.0:*

logstash.yml如下

node.name: server-b
path.data: /var/lib/logstash
http.host: "0.0.0.0"

modules:
  - name: netflow
    var.input.udp.port: 9995 # Inbound connections
    var.elasticsearch.hosts: "<ip address>:9200"
    var.kibana.host: "<ip address>:5601"

path.logs: /var/log/logstash

检查/var/log/logstash/logstash-plain.log，我看到的唯一警告是有关elasticsearch的版本大于版本6的信息，因此不会使用type 确定文档类型。

[2018-07-06T12:58:13,771][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-07-06T12:58:13,817][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-07-06T12:58:17,599][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"module-netflow", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-07-06T12:58:17,733][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://<ip address>:9200/]}}
[2018-07-06T12:58:17,734][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://<ip address>:9200/, :path=>"/"}
[2018-07-06T12:58:17,784][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://<ip address>:9200/"}
[2018-07-06T12:58:17,810][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-07-06T12:58:17,810][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-07-06T12:58:17,811][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//<ip address>:9200"]}
[2018-07-06T12:58:18,088][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-07-06T12:58:18,101][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-ASN.mmdb"}
[2018-07-06T12:58:18,102][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-07-06T12:58:18,103][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-ASN.mmdb"}
[2018-07-06T12:58:18,103][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-07-06T12:58:18,103][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-ASN.mmdb"}
[2018-07-06T12:58:18,104][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-07-06T12:58:18,104][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-ASN.mmdb"}
[2018-07-06T12:58:18,120][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:9995"}
[2018-07-06T12:58:18,126][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"module-netflow", :thread=>"#<Thread:0x16700849@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}
[2018-07-06T12:58:18,131][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:9995", :receive_buffer_bytes=>"212992", :queue_size=>"2000"}
[2018-07-06T12:58:18,135][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["module-netflow"]}

ElasticSearch正在运行，并且正在从packetbeat和filebeat接收数据，/ var / log / elasticsearch / elasticsearch.log中没有任何内容可以暗示Elasticsearch的任何错误。但是，elasticsearch没有Netflow的索引模式。另一方面，基巴纳则做到了。

因此，服务器B上的logstash正在侦听0.0.0.0:9995，端口9995已打开并正在接收来自服务器A的数据包，但是logstash无法识别这些数据包。我的假设是服务器B忽略了它们，因为目标IP地址是服务器A的地址。听起来正确吗？如果是这样，这附近是否存在？

是否有更好的方法将重复的数据包从服务器A转发到服务器B，并让logstash读取它们？

不幸的是，无法将另一个netflow导出器目标添加到路由器配置中。

Answer 1

我会回答我自己的问题。

服务器B确实忽略了netflow数据，因为它无法识别IP地址。我添加了服务器A的IP地址作为回送接口，它可以按预期工作。

这可能不是最佳解决方案，在生产环境中应避免使用，但出于测试目的，应该没问题。

Logstash Netflow模块正在侦听，但不读取数据包

1 个答案: