Django memcache低级api提供的性能低于缓存中间件

Question

这是我第一次在这里发帖提问，请忽略与标准格式的偏差......

这是我的设置。 Nginx - ＆gt; Uwsgi - ＆gt; Django - ＆gt; （Memcached / Mysql）

我在Django中有一组返回json响应的API。 之前我使用memcached 2部分中间件来进行自动缓存。现在为了获得更多控制，我已经转向低级API。我使用celery创建pickle数据集并每天在memcached中插入一次数据，My Apis从memcached中获取数据。

我在我的生产环境中使用NGINX和Uwsgi（16名工作人员），并且使用2部分中间件方法获得大约8K请求的速度。只有我的第一个请求过去很慢。 但是对于较低级别的缓存，我获得了1/10的速度。

最初我认为这是由于额外的中间件和自定义代码被执行，但我也修剪了它。我从泡菜搬到cPickle，但我不认为这是瓶颈。我正在使用apache基准来进行测试。

我为中间件方法和低级缓存api附加了Memcached（-vv）日志。 如果有人能指出我正确的方向........谢谢

我看到了Django的catche-middleware代码。他们似乎在缓存中保存django httpresponse对象。中间件也在内部调用低级apis。我也尝试在缓存中保存httpresponse，但没有变化......

日志

<pre>

MIDDLEWARE approach memcached LOGS
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 END
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 END
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<34 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 END
<38 new auto-negotiating client connection
<42 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 END
42: Client using the ascii protocol
<42 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 END
>37 END
<41 new auto-negotiating client connection
<33 connection closed.
<36 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 END
41: Client using the ascii protocol
<41 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 END
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 END
<38 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<31 connection closed.
<36 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<39 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<35 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 END
<33 connection closed.
<30 connection closed.
<42 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 END
<37 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 END
<35 connection closed.
<39 connection closed.
<32 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<30 new auto-negotiating client connection
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<38 connection closed.
<32 connection closed.
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<33 connection closed.
<37 connection closed.
<31 connection closed.
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<34 connection closed.
<42 connection closed.
<41 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 END
<40 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 END
<28 connection closed.
<30 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<30 new auto-negotiating client connection
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<40 connection closed.

</pre>


----------


----------


LOW level logs
<pre>
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||
>32 END
<36 connection closed.
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:ls|1|0|10|||
>36 sending key :1:ls|1|0|10|||
>36 END
<40 connection closed.
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:ls|1|0|10|||
>40 sending key :1:ls|1|0|10|||
>40 END
<34 connection closed.
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:ls|1|0|10|||
>34 sending key :1:ls|1|0|10|||
>34 END
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:ls|1|0|10|||
>29 sending key :1:ls|1|0|10|||
>29 END
<35 connection closed.
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:ls|1|0|10|||
>35 sending key :1:ls|1|0|10|||
>35 END
<37 connection closed.
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:ls|1|0|10|||
>37 sending key :1:ls|1|0|10|||
>37 END
<41 connection closed.
<41 new auto-negotiating client connection
41: Client using the ascii protocol
<41 get :1:ls|1|0|10|||
>41 sending key :1:ls|1|0|10|||
>41 END
<39 connection closed.
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:ls|1|0|10|||
>39 sending key :1:ls|1|0|10|||
>39 END
<38 connection closed.
<38 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:ls|1|0|10|||
>38 sending key :1:ls|1|0|10|||
>38 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:ls|1|0|10|||
>30 sending key :1:ls|1|0|10|||
>30 END
<31 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:ls|1|0|10|||
>31 sending key :1:ls|1|0|10|||
>31 END
<42 connection closed.
<42 new auto-negotiating client connection
42: Client using the ascii protocol
<42 get :1:ls|1|0|10|||
>42 sending key :1:ls|1|0|10|||
>42 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:ls|1|0|10|||
>28 sending key :1:ls|1|0|10|||
>28 END
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<34 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||

</pre>

Answer 1

经过一周的尝试和测试（感谢我的团队），我意识到Django低级Apis有利于获得优于数据库获取的优势但不适用于严重的缓存性能。 Django中间件类仅在中间件堆栈吞吐量服务器进程中实例化一次。正如Django official docs和django book中提到的那样。

在使用低级api时，启动Memcache的连接可能花费了太多时间。 最后，我继承了FetchfromCache middleware并创建了我自己的自定义中间件，它提供了与Django的2部分中间件相同的性能。剥离不需要的中间件也有助于提升性能。