Question

我试图在ES中索引一些数据并且我收到了内存异常：

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.elasticsearch.common.jackson.core.util.BufferRecycler.balloc(BufferRecycler.java:155)
    at org.elasticsearch.common.jackson.core.util.BufferRecycler.allocByteBuffer(BufferRecycler.java:96)
    at org.elasticsearch.common.jackson.core.util.BufferRecycler.allocByteBuffer(BufferRecycler.java:86)
    at org.elasticsearch.common.jackson.core.io.IOContext.allocWriteEncodingBuffer(IOContext.java:152)
    at org.elasticsearch.common.jackson.core.json.UTF8JsonGenerator.<init>(UTF8JsonGenerator.java:123)
    at org.elasticsearch.common.jackson.core.JsonFactory._createUTF8Generator(JsonFactory.java:1284)
    at org.elasticsearch.common.jackson.core.JsonFactory.createGenerator(JsonFactory.java:1016)
    at org.elasticsearch.common.xcontent.json.JsonXContent.createGenerator(JsonXContent.java:68)
    at org.elasticsearch.common.xcontent.XContentBuilder.<init>(XContentBuilder.java:96)
    at org.elasticsearch.common.xcontent.XContentBuilder.builder(XContentBuilder.java:77)
    at org.elasticsearch.common.xcontent.json.JsonXContent.contentBuilder(JsonXContent.java:38)
    at org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:122)
    at org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder(XContentFactory.java:49)
    at EsController.importProductEs(EsController.java:60)
    at Parser.fromCsvToJson(Parser.java:120)
    at CsvToJsonParser.parseProductFeeds(CsvToJsonParser.java:43)
    at MainParser.main(MainParser.java:49)

这是我实例化ES客户端的方式：

System.out.println("Elastic search client is instantiated");
Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "elasticsearch_brew").build();
client = new TransportClient(settings);
String hostname = "localhost";
int port = 9300; 
((TransportClient) client).addTransportAddress(new InetSocketTransportAddress(hostname, port));     
bulkRequest = client.prepareBulk();

然后我运行批量请求：

// for each product in the list, we need to include the fields in the bulk request 
for(HashMap<String, String> productfields : products)
        try {
            bulkRequest.add(client.prepareIndex(index,type,productfields.get("Product_Id"))
                    .setSource(jsonBuilder()
                                .startObject()
                                    .field("Name",productfields.get("Name") )
                                    .field("Quantity",productfields.get("Quantity"))
                                    .field("Make", productfields.get("Make"))
                                    .field("Price", productfields.get("Price"))
                                .endObject()
                              )
                    );                  

        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
//execute the bulk request
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures()) {
    // process failures by iterating through each bulk response item
}

我正在尝试索引各个商店的产品。每个商店都是不同的指数。当我到达包含约60000种产品的第6家商店时，我得到了上述例外情况。我将批量请求拆分为10000块，试图避免内存不足问题。我无法理解瓶颈究竟在哪里。如果我以某种方式刷新批量请求或重新启动客户端会有帮助吗？我看过类似的帖子但不适用于我。

修改

每当我处理新的批量请求时，我立即启动新客户端，那么我就不会出现内存不足异常。但每次实例化一个新的客户端似乎都没有。

谢谢

Answer 1

所以我弄清楚出了什么问题。

每个新的批量请求都会累加到前一个批量请求，最终导致内存不足。

所以现在在我开始新的批量请求之前，我运行了 bulkRequest = client.prepareBulk(); 它会刷新上一个请求。

谢谢你们的评论

ElasticSearch内存不足

1 个答案: