Question

我们计划在其中一个项目中使用ElasticSearch。目前，我们正在使用我们的数据测试ElasticSearch 5.0.1。我们面临的一个问题是，当我们正在从我们的MySQL表批量上传到弹性搜索时我们正在获得错误......

java.lang.IllegalArgumentException: Limit of total fields [1000] in index [shopfront] has been exceeded
at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:482) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:343) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:277) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:323) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:241) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:555) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:896) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:451) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) ~[elasticsearch-5.0.1.jar:5.0.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

我们使用PHP作为弹性搜索客户端来进行从MySQL到Elastic的批量上传。在做了一些谷歌搜索后，我得到了这条信息 - https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821

某处我还读到使用“index.mapping.total_fields.limit”将解决这个问题。但是，无法理解如何在我的PHP代码中使用它。这是我的PHP代码。

$params = ['body' => []];

$i = 1;
foreach ($productsList as $key => $value) {

    $params['body'][] = [
        'index' => [
            '_index' => 'shopfront',
            '_type' => 'products'
        ],
        'settings' => ['index.mapping.total_fields.limit' => 3000]
    ];

    $params['body'][] = [
        'product_displayname' => $value['product_displayname'],
        'product_price' => $value['product_price'],
        'popularity' => $value['popularity'],
        'lowestcomp_price' => $value['lowestcomp_price']
    ];

    // Every 1000 documents stop and send the bulk request
    if ($i % 1000 == 0) {
        $responses = $client->bulk($params);

        // erase the old bulk request
        $params = ['body' => []];

        // unset the bulk response when you are done to save memory
        unset($responses);
    }

    $i++;
}

// Send the last batch if it exists
if (!empty($params['body'])) {
    $responses = $client->bulk($params);
}

注意 - 我在Elasticsearch 2.4.1＆amp;中使用了相同的代码。它的工作正常。

Answer 1

在ES 5中，ES人员决定限制映射类型可以包含的字段数，以防止映射爆炸。正如您所注意到的，该限制已设置为每个映射1000个字段，但您可以通过在索引创建时或updating the index settings指定index.mapping.total_fields.limit设置来提升该限制以满足您的需求，例如这样：

curl -XPUT 'localhost:9200/shopfront/_settings' -d '
{
    "index.mapping.total_fields.limit": 3000
}'

请注意，你还需要问问自己，拥有那么多领域是否是一件好事。你需要它们吗？你可以结合一些吗？等等

Answer 2

此功能已在此github issue中确定。解决这个问题的两种方法：

您可以在创建索引时指定更大的值：

PUT test
{
  "shopfront": {
    "index.mapping.total_fields.limit": 2000,
    "number_of_shards": 5,
    "number_of_replicas": 2
  },
  "mappings": {
    ...
  }
}

或者，如果您想增加现有索引的限制：

PUT shopfront/_settings
{
  "index.mapping.total_fields.limit": 2000
}

使用PHP进行Elasticsearch批量上传错误 - 已超出索引中总字段数[1000]的限制

2 个答案: