Elasticsearch批量不再索引文档,但没有错误

时间:2015-06-26 14:50:23

标签: elasticsearch bulkinsert

6.0

运行ES 1.6.0

我有一个批量索引工作,以前工作正常。我想在新索引中批量索引一些文档。我发了大约100K +文件,但只有几千个被索引。

我检查日志,ES节点上没有记录特定错误,我的JAVA客户端没有返回任何错误。

我还看了Marvel Bulk Thread Pool没有拒绝。队列大小为0。

我有20TB的磁盘空间,其他索引使用13TB。所以我的磁盘水印低于其他任何东西?

这是膨胀代码......

        BulkRequestBuilder bulkRequest = client.prepareBulk();
        final Context ctx = getVertx().currentContext();

        for(int i = 0; i < documents.size(); i++)
        {   
            final JsonObject obj = documents.get(i);
            final JsonObject indexable = new JsonObject()

            .putString("action", "index")
            .putString("_index", obj.getString("index"))
            .putString("_type", obj.getString("type"))
            .putString("_id", obj.getString("id"))
            .putString("_route", obj.getString("routing"))
            .putObject("_source", obj);                                     

            final String index = getRequiredIndex(indexable, message);
            if (index == null) {
                return;
            }

            // type is optional
            String type = indexable.getString(CONST_TYPE);
            ;

            JsonObject source = indexable.getObject(CONST_SOURCE);
            if (source == null) {
                sendError(message, CONST_SOURCE + " is required");
                return;
            }

            // id is optional
            String id = indexable.getString(CONST_ID);
            String route = indexable.getString(CONST_ROUTE);

            IndexRequestBuilder builder = client.prepareIndex(index, type, id).setSource(source.encode());

            if(!route.isEmpty())
                builder.setRouting(route);              

            bulkRequest.add(builder);
        }

        bulkRequest.execute(new ActionListener<BulkResponse>(){
            @Override
            public void onResponse(BulkResponse resp) {


                message.reply(new JsonObject().putString("status", "Took: " + resp.getTookInMillis() + ", Indexed:" + documents.size() + "," + resp.getItems().length + ", Failed: " + resp.hasFailures()));
            }

            @Override
            public void onFailure(Throwable t) {
                ctx.runOnContext(new Handler<Void>() {
                    @Override
                    public void handle(Void event) {
                        sendError(message,
                                "Index error: " + t.getMessage(),
                                new RuntimeException(t));
                    }
                });
            }
        });

0 个答案:

没有答案