如何将Elasticsearch批量请求发送到Amazon SGS队列

时间:2019-07-31 15:39:33

标签: elasticsearch amazon-sqs

我正在使用org.elasticsearch.action.bulk.BulkRequest通过Amazon sqs标准队列进行发送。

我能够将请求发送到sqs队列,并且可以通过命令行在队列中看到批量请求消息。

当我尝试接收消息并使用org.elasticsearch.action.bulk.BulkRequest解析返回异常时

这是制作人

public void sendMessage(){
        ObjectMapper mapper = new ObjectMapper();
        long startIndex = 0;
        long batchSize = queueConfiguration.getBatchSize();
        String index = getIndex();
        int count = 1;
        log.info("Batch size is : " + batchSize);
        while(true){
            long startTime = System.currentTimeMillis();
            List<Order> orderList = serviceImpl.fetchBatch(startIndex, batchSize);

            BulkRequest bulkRequest = new BulkRequest();
            for (Order order : orderList) {
                Map<String, Object> documentMapper = mapper.convertValue(order, Map.class);
                IndexRequest indexRequest = new IndexRequest(index).source(documentMapper);
                indexRequest.id(count + "");
                count++;
                bulkRequest.add(indexRequest);
            }

            try{
                SendMessageRequest messageRequest = new SendMessageRequest()
                        .withQueueUrl(queueConfiguration.getSqsEndPointUrl())
                        .withMessageBody(mapper.writeValueAsString(bulkRequest))
                        .withDelaySeconds(10);
                amazonSQS.sendMessage(messageRequest);
            } catch (Exception ex){
                log.error("Exception encountered while putting orderlist in queue : " + ex);
            }

            if(orderList.size() < batchSize){
                break;
            }
            startIndex = startIndex + batchSize;
            long endTime = System.currentTimeMillis();
            log.info("Time taken to fetch : " + startIndex + " is : " + (endTime - startTime) + "ms");
        }
    }

消费者是

@SqsListener(value = "queue-name")
    public void indexOrders(String message) {
        long startTime = System.currentTimeMillis();
        ObjectMapper mapper = new ObjectMapper();

        try{
            BulkRequest bulkRequest = mapper.readValue(message, BulkRequest.class);
            client.bulk(bulkRequest, RequestOptions.DEFAULT);
        } catch (Exception ex){
            log.error("Exception while parsing bulk request" , ex);
        }
        log.info(message);

        long endTime = System.currentTimeMillis();
        log.info("Time taken to process message is : " + (endTime - startTime));
    }

我遇到异常,请在(BulkRequest bulkRequest = mapper.readValue(message,BulkRequest.class);)行中检查使用者代码 com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "description" (class org.elasticsearch.action.bulk.BulkRequest), not marked as ignorable (2 known properties: "refreshPolicy", "parentTask"])

任何帮助将不胜感激。

0 个答案:

没有答案