Question

我正在使用以下flume 1.7代理配置从Kafka 0.9.0.1主题流式传输数据，并将数据发送到使用目录版本v0.5.0中的ES在Rancher上设置的ElasticSearch。

agent.sources  = flume_test
agent.channels = kafka_test_channel
agent.sinks =   ElasticSearchSink

agent.sources.flume_test.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.flume_test.zookeeperConnect = stage-kafka01.stage:2181,stage-kafka02.stage:2181,stage-kafka03.stage:2181
agent.sources.flume_test.topic = hrzn_stage1_test
agent.sources.flume_test.groupId = flume_kafka_stage1_test
agent.sources.flume_test.channels = kafka_test_channel
agent.sources.flume_test.spoolDir = /var/log/spoolDir/auth
agent.sources.flume_test.interceptors = i1
agent.sources.flume_test.interceptors.i1.type = com.cpm.interceptors.TestInterceptor$Builder

agent.channels.kafka_test_channel.type = file
agent.channels.kafka_test_channel.checkpointDir =  /dev/shm/flume/checkpointsDir/auth
agent.channels.kafka_test_channel.dataDirs = /dev/shm/flume/dataDir/auth
agent.channels.kafka_test_channel.checkpointInterval = 10
agent.channels.kafka_test_channel.capacity = 10000
agent.channels.kafka_test_channel.transactionCapacity = 1000

agent.sinks.ElasticSearchSink.type = elasticsearch
agent.sinks.ElasticSearchSink.hostNames = 10.42.242.78:9300
agent.sinks.ElasticSearchSink.indexName = auth
agent.sinks.ElasticSearchSink.indexType = logs
agent.sinks.ElasticSearchSink.clusterName = elasticsearch
agent.sinks.ElasticSearchSink.batchSize = 500
agent.sinks.ElasticSearchSink.ttl = 5d
agent.sinks.ElasticSearchSink.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer 
agent.sinks.ElasticSearchSink.channel = kafka_test_channel

10.42.242.78是我从日志中提取的发布地址之一，即

publish_address {10.42.242.78:9300}, bound_addresses {10.42.242.78:9300}, {172.17.0.5:9300}

在ES之上，我使用的是elasticsearch-kopf，我创建了一个具有以下设置的索引

{
  "number_of_shards": "1",
  "number_of_replicas": "1",
  "mappings": {
    "default": {
      "properties": {
        "uuid": {
          "type": "string"
        },
        "brandId": {
          "type": "string"
        },
        "creationDate": {
          "type": "string"
        }
      }
    }
  }
}

Flume连接到发布端点，但我根本无法在ES中看到数据。我也试图使用索引进行搜索，但没有任何结果。

从Flume发送的消息示例可以是

{"uuid":"12345","brandId":"12345","creationDate":"2017-09-13T20:32:15.660","payload":"test"}

我错过了什么？

将数据从Apache Flume发送到ElasticSearch

0 个答案: