elasticsearch:如何解释日志文件(集群变为黄色状态)?

时间:2015-11-24 16:53:02

标签: elasticsearch

CentOS上的Elasticsearch 1.7.2,8GB RAM,2节点集群。

我们在此处发布了整个日志:http://pastebin.com/zc2iG2q4

当我们查看/ _cluster / health时,我们会看到2个未分配的分片:

{
  "cluster_name" : "elasticsearch-prod",
  "status" : "yellow",    <--------------------------
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 5,
  "active_shards" : 8,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 2,    <--------------------------
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0

在日志中,我们看到:

marking and sending shard failed due to [failed to create shard]
java.lang.OutOfMemoryError: Java heap space

还有其他错误。

我们唯一与内存相关的配置值是:

indices.fielddata.cache.size:  75%

我们期待:

  • 更完整地了解日志
  • 了解我们现在需要采取什么行动(恢复)并在将来阻止它

其他详情:

1)ES_HEAP_SIZE是库存,没有变化。 (此外,环顾四周,目前还不清楚最好改变它...... /etc/init.d/elasticsearch?)

2)我们的jvm统计数据如下。 (请注意,作为测试,我修改了"/etc/init.d/elasticsearch"并添加了导出ES_HEAP_SIZE=4g [代替现有的"export ES_HEAP_SIZE"行]并重新启动ES ....比较两个相同的节点,一个包含已更改的elasticsearch文件和一个库存,下面的值看起来相同)

  "jvm" : {
    "timestamp" : 1448395039780,
    "uptime_in_millis" : 228297,
    "mem" : {
      "heap_used_in_bytes" : 81418872,
      "heap_used_percent" : 7,
      "heap_committed_in_bytes" : 259522560,
      "heap_max_in_bytes" : 1037959168,
      "non_heap_used_in_bytes" : 50733680,
      "non_heap_committed_in_bytes" : 51470336,
      "pools" : {
        "young" : {
          "used_in_bytes" : 52283368,
          "max_in_bytes" : 286326784,
          "peak_used_in_bytes" : 71630848,
          "peak_max_in_bytes" : 286326784
        },
        "survivor" : {
          "used_in_bytes" : 2726824,
          "max_in_bytes" : 35782656,
          "peak_used_in_bytes" : 8912896,
          "peak_max_in_bytes" : 35782656
        },
        "old" : {
          "used_in_bytes" : 26408680,
          "max_in_bytes" : 715849728,
          "peak_used_in_bytes" : 26408680,
          "peak_max_in_bytes" : 715849728
        }
      }
    },
    "threads" : {
      "count" : 81,
      "peak_count" : 81
    },
    "gc" : {
      "collectors" : {
        "young" : {
          "collection_count" : 250,
          "collection_time_in_millis" : 477
        },
        "old" : {
          "collection_count" : 1,
          "collection_time_in_millis" : 22
        }
      }
    },
    "buffer_pools" : {
      "direct" : {
        "count" : 112,
        "used_in_bytes" : 20205138,
        "total_capacity_in_bytes" : 20205138
      },
      "mapped" : {
        "count" : 0,
        "used_in_bytes" : 0,
        "total_capacity_in_bytes" : 0
      }
    }
  },

1 个答案:

答案 0 :(得分:0)

解决。

此处的关键是错误"java.lang.OutOfMemoryError: Java heap space"

另一天,来自ES docs的另一个宝石:

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

说(强调我的):

Elasticsearch的默认安装配置为1 GB堆。 对于几乎每个部署,此数字都太小。如果您使用的是默认堆值,则您的群集可能配置不正确。

分辨率:

修改:/etc/sysconfig/elasticsearch

设置ES_HEAP_SIZE=4g //此系统具有8GB RAM

重启ES

并且tada ....未分配的分片被神奇地分配,并且群集变为绿色。