我正在阅读this article,其中提到在redis中存储数十亿个密钥将使用17GB的内存。然而,当切换到哈希时,每个以1k为单位分块(例如:HSET "mediabucket:1155" "1155315" "939"
)允许它们以5GB存储1M,这是一个非常大的节省。
我读过redis memory-optimization,但我不太明白其中的区别。它说HGET不是很好O(1)但足够接近,并且在使用hsets时会提到更多的CPU使用率。我不明白为什么会有更多的CPU使用(确定空间的交易时间。但是如何/什么?)。它提到了“编码”,但没有提到它们如何编码。
它也只提到了字符串,但我不知道只有字符串的含义。它是哈希字段吗?这是否意味着哈希字段?我在HSET中没有看到任何相关内容。究竟会编码什么,为什么编码比使用SET更有效?
如何HSET "mediabucket:1155" "1155315" "939"
比SET "mediabucket:1155315" "939"
更有效率? SET中的数据较少(使用1155315和1155而不是1155315)。我个人会尝试使用二进制密钥,但我不认为这与HSET为什么效率更高有关。
编辑:
Cross也发布在redis-db邮件列表上:https://groups.google.com/d/topic/redis-db/90K3UqciAx0/discussion
答案 0 :(得分:19)
小哈希对象被编码为ziplists,具体取决于hash-max-ziplist-entries和hash-max-ziplist-value参数的值。这是简单的数据序列化。
ziplist定义如下(从Redis源代码中提取):
/* The ziplist is a specially encoded dually linked list that is designed
* to be very memory efficient. It stores both strings and integer values,
* where integers are encoded as actual integers instead of a series of
* characters. It allows push and pop operations on either side of the list
* in O(1) time. However, because every operation requires a reallocation of
* the memory used by the ziplist, the actual complexity is related to the
* amount of memory used by the ziplist.
*
* ----------------------------------------------------------------------------
*
* ZIPLIST OVERALL LAYOUT:
* The general layout of the ziplist is as follows:
* <zlbytes><zltail><zllen><entry><entry><zlend>
*
* <zlbytes> is an unsigned integer to hold the number of bytes that the
* ziplist occupies. This value needs to be stored to be able to resize the
* entire structure without the need to traverse it first.
*
* <zltail> is the offset to the last entry in the list. This allows a pop
* operation on the far side of the list without the need for full traversal.
*
* <zllen> is the number of entries.When this value is larger than 2**16-2,
* we need to traverse the entire list to know how many items it holds.
*
* <zlend> is a single byte special value, equal to 255, which indicates the
* end of the list.
*
* ZIPLIST ENTRIES:
* Every entry in the ziplist is prefixed by a header that contains two pieces
* of information. First, the length of the previous entry is stored to be
* able to traverse the list from back to front. Second, the encoding with an
* optional string length of the entry itself is stored.
*
* The length of the previous entry is encoded in the following way:
* If this length is smaller than 254 bytes, it will only consume a single
* byte that takes the length as value. When the length is greater than or
* equal to 254, it will consume 5 bytes. The first byte is set to 254 to
* indicate a larger value is following. The remaining 4 bytes take the
* length of the previous entry as value.
*
* The other header field of the entry itself depends on the contents of the
* entry. When the entry is a string, the first 2 bits of this header will hold
* the type of encoding used to store the length of the string, followed by the
* actual length of the string. When the entry is an integer the first 2 bits
* are both set to 1. The following 2 bits are used to specify what kind of
* integer will be stored after this header. An overview of the different
* types and encodings is as follows:
*
* |00pppppp| - 1 byte
* String value with length less than or equal to 63 bytes (6 bits).
* |01pppppp|qqqqqqqq| - 2 bytes
* String value with length less than or equal to 16383 bytes (14 bits).
* |10______|qqqqqqqq|rrrrrrrr|ssssssss|tttttttt| - 5 bytes
* String value with length greater than or equal to 16384 bytes.
* |11000000| - 1 byte
* Integer encoded as int16_t (2 bytes).
* |11010000| - 1 byte
* Integer encoded as int32_t (4 bytes).
* |11100000| - 1 byte
* Integer encoded as int64_t (8 bytes).
* |11110000| - 1 byte
* Integer encoded as 24 bit signed (3 bytes).
* |11111110| - 1 byte
* Integer encoded as 8 bit signed (1 byte).
* |1111xxxx| - (with xxxx between 0000 and 1101) immediate 4 bit integer.
* Unsigned integer from 0 to 12. The encoded value is actually from
* 1 to 13 because 0000 and 1111 can not be used, so 1 should be
* subtracted from the encoded 4 bit value to obtain the right value.
* |11111111| - End of ziplist.
*
* All the integers are represented in little endian byte order.
*/
哈希对象中的每个项目都表示为ziplist中的键/值对(2个连续的条目)。键和值都可以存储为简单字符串或整数。这种格式在内存中更紧凑,因为它节省了许多实现动态数据结构所需的指针(每个8字节)(如真正的哈希表)。
缺点是HSET / HGET操作在ziplist上应用时实际上是O(N)。这就是为什么ziplist必须保持小。当ziplist数据适合L1 CPU缓存时,相应的算法尽管线性复杂,但速度足够快。
您可能需要参考以下链接以获取更多信息:
Redis 10x more memory usage than data
Redis Data Structure Space Requirements
这些答案涉及其他数据结构(如集合,列表或有序集合),但它们的概念完全相同。