运行一段时间后,memcached失败

时间:2016-10-16 01:07:21

标签: python django docker memcached

我正在使用带有大量请求的memcached的Django。 起初memcached工作正常,但是,一段时间后它会不时地开始出错。错误消息也不一致。花了很多时间后,我甚至无法找出错误究竟是什么意思,也不知道如何解决它。

设定:

  • Django +默认的memcached后端
  • 使用Docker启动memcached: docker up -d memcached -m 3072m -I 10m -c 4096
  • memcached使用自己的专用主机
  • 主机规格:4 CPU + 3.5GB内存
  • swappiness = 0
  • 在Google计算引擎上运行

统计

 stats
    STAT pid 1
    STAT uptime 38718
    STAT time 1476576775
    STAT version 1.4.31
    STAT libevent 2.0.21-stable
    STAT pointer_size 64
    STAT rusage_user 73.432000
    STAT rusage_system 565.536000
    STAT curr_connections 10
    STAT total_connections 346393
    STAT connection_structures 4070
    STAT reserved_fds 20
    STAT cmd_get 710031
    STAT cmd_set 373473
    STAT cmd_flush 0
    STAT cmd_touch 0
    STAT get_hits 681898
    STAT get_misses 28133
    STAT get_expired 1295
    STAT get_flushed 0
    STAT delete_misses 0
    STAT delete_hits 0
    STAT incr_misses 0
    STAT incr_hits 0
    STAT decr_misses 0
    STAT decr_hits 0
    STAT cas_misses 0
    STAT cas_hits 0
    STAT cas_badval 0
    STAT touch_hits 0
    STAT touch_misses 0
    STAT auth_cmds 0
    STAT auth_errors 0
    STAT bytes_read 294333584541
    STAT bytes_written 294114029877
    STAT limit_maxbytes 3221225472
    STAT accepting_conns 1
    STAT listen_disabled_num 8604
    STAT time_in_listen_disabled_us 207162898
    STAT threads 4
    STAT conn_yields 0
    STAT hash_power_level 16
    STAT hash_bytes 524288
    STAT hash_is_expanding 0
    STAT malloc_fails 0
    STAT log_worker_dropped 0
    STAT log_worker_written 0
    STAT log_watcher_skipped 0
    STAT log_watcher_sent 0
    STAT bytes 80025379
    STAT curr_items 4616
    STAT total_items 373473
    STAT expired_unfetched 20032
    STAT evicted_unfetched 0
    STAT evictions 0
    STAT reclaimed 21994
    STAT crawler_reclaimed 0
    STAT crawler_items_checked 0
    STAT lrutail_reflocked 1229
    END

不同的错误消息

Exception Value: error 31 from memcached_get(:1:a7735b06da00d2e6991920299c31): (14006136669712) A TIMEOUT OCCURRED, host: 10.140.0.16:11211 -> libmemcached/get.cc:314

error 37 from memcached_set: SUCCESS

error 47 from memcached_get(:1:throttle_naive_9885206_None_): (140085126390144) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, host: 10.140.0.16:11211 -> libmemcached/get.cc:314

请注意

  • 在所有情况下,似乎memcached docker仍然可以正常运行 stats仍然有效,并非所有请求都会生成错误
  • memcached主机的CPU使用率仅为1%
  • 我将memcached实例与我的前端服务放在同一主机上时遇到了同样的问题。但是,同样的问题。
编辑:由于我无法解决此问题,我在同一主机上切换到了Redis。虽然我没有在第一时间做到正确,但Redis提供了全面的错误消息,因此我可以轻松解决它们。现在我的缓存运行稳定。

0 个答案:

没有答案