我使用Elasticsearch Connector作为接收器,将数据插入Elasticsearch(请参阅:https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/elasticsearch.html)。
但是,我没有找到任何连接器来从Elasticsearch获取数据作为源。
在Flink管线中是否有任何连接器或示例将Elasticsearch文档用作源?
此致
阿里
答案 0 :(得分:0)
我不知道Flink的显式ES来源。我确实看到一个用户在谈论将elasticsearch-hadoop用作Flink的HadoopInputFormat
,但我不知道这是否对他们有用(请参阅their code)。
答案 1 :(得分:0)
我最终定义了从ElasticSearch函数读取的简单内容
public static class ElasticsearchFunction
extends ProcessFunction<MetricMeasurement, MetricPrediction> {
public ElasticsearchFunction() throws UnknownHostException {
client = new PreBuiltTransportClient(settings)
.addTransportAddress(new TransportAddress(InetAddress.getByName("YOUR_IP"), PORT_NUMBER));
}
@Override
public void processElement(MetricMeasurement in, Context context, Collector<MetricPrediction> out) throws Exception {
MetricPrediction metricPrediction = new MetricPrediction();
metricPrediction.setMetricId(in.getMetricId());
metricPrediction.setGroupId(in.getGroupId());
metricPrediction.setBucket(in.getBucket());
// Get the metric measurement from Elasticsearch
SearchResponse response = client.prepareSearch("YOUR_INDEX_NAME")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(QueryBuilders.termQuery("YOUR_TERM", in.getMetricId())) // Query
.setPostFilter(QueryBuilders.rangeQuery("value").from(0L).to(50L)) // Filter
.setFrom(0).setSize(1).setExplain(true)
.get();
SearchHit[] results = response.getHits().getHits();
for(SearchHit hit : results){
String sourceAsString = hit.getSourceAsString();
if (sourceAsString != null) {
ObjectMapper mapper = new ObjectMapper();
MetricMeasurement obj = mapper.readValue(sourceAsString, MetricMeasurement.class);
obj.getMetricId();
metricPrediction.setPredictionValue(obj.getValue());
}
}
out.collect(metricPrediction);
}
}
答案 2 :(得分:0)
Hadoop 兼容性 + Elasticsearch Hadoop
https://github.com/cclient/flink-connector-elasticsearch-source