Question

我将数据存储到Apache Kafka中。然后我能够使用Apache Storm spout使用数据并处理数据。现在我想将处理过的数据导出到弹性搜索中。

Answer 1

ElasticSearch provides Java client api available in maven repository.

If you already implemented pulling data from kafka to storm, all you have to do is implement one bolt that sends indexing request of that log to the elastic search.

Here I'm talking in traditional topology perspective.

For example, in your prepare method implementation, you create a transport client like this.

Client client = new TransportClient()
    .addTransportAddress(new InetSocketTransportAddress("host1", 9300))
    .addTransportAddress(new InetSocketTransportAddress("host2", 9300));

And in your execute method implementation, you send indexing request like this.

String json = "{" +
    "\"user\":\"kimchy\"," +
    "\"postDate\":\"2013-01-30\"," +
    "\"message\":\"trying out Elasticsearch\"" +
"}";

IndexResponse response = client.prepareIndex("twitter", "tweet")
    .setSource(json)
    .execute()
    .actionGet();

For more information, refer to http://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html.

Answer 2

您可以使用ES Storm集成：

https://www.elastic.co/guide/en/elasticsearch/hadoop/current/storm.html

或者将您的数据写回Kafka并使用LogStash来使用此队列并写入elasticsearch。

如何将数据从风暴导出到弹性搜索？

2 个答案: