Question

我们目前有一个配置如下的logstash：

input {
  rabbitmq {
    id => "xxxxxxxxxxxx"
    host => "xxx.xxx.xxx.xxx"
    exchange => "xxxxxx_xxxxxxx_xxxxx"
    key => "xxxxxx_xxxxxxx_xxxxx"
    queue => "xxxxxx_xxxxxxx_xxxxx"
    user => "shipper"
    password => "xxxxxxxxxxxxx"
    durable => true
    type => "xxxxxx_xxxxxxx_xxxxx"
    threads => 1
    prefetch_count => 100
    connection_timeout => 10000
    vhost => "vhost_xxxxxxxx"
  }
}
output {
  elasticsearch {
    hosts => ["xxx.xxx.xxx.xxx:9200","xxx.xxx.xxx.xxx:9200","xxx.xxx.xxx.xxx:9200"]
    index => "xxxxxx_xxxxxxx_xxxxx"
    document_type => "xxxxxx_xxxxxxx_xxxxx"
    timeout => 30000
  }
  elasticsearch {
    hosts => ["load_balanced_host:80"]
    index => "xxxxxx_xxxxxxx_xxxxx"
    document_type => "xxxxxx_xxxxxxx_xxxxx"
    timeout => 30000
  }
}

第一个输出（具有3 ip的输出）是新集群（我们要迁移到的集群）。第二个是由Amazon Elasticsearch服务提供的负载平衡集群。它们具有相同的映射。它们都是同一版本（2.3）。

问题是：新群集丢失了消息（或者，idk无法传递消息）。让我们以这种情况为例：

    1 - MSG_A, MSG_B and MSG_C arrives in the rabbitMq;
    2 - Logstash processes MSG_A, MSG_B and MSG_C;
    3 - Logstash logs shows nothing anormal;
    4 - Checking the old elasticsearch cluster all the messages are there (MSG_A, MSG_B and MSG_C);
    5 - Checking the new elasticsearch cluster there's messages missing.

任何人都知道我可以在哪里开始调查，或者为什么会这样？

谢谢！

试图切换输出顺序（新群集优先/旧群集优先，旧群集优先/新群集优先），并且没有任何变化-在新群集中仍然丢失消息。

在logstash中使用多个输出时丢失数据

0 个答案: