MemSQL需要15GB内存才能获得10MB数据

时间:2017-03-27 05:53:00

标签: memory memsql

我已按照以下方式使用以下资源安装了memsql 5.1.2。

  

Google云端服务器

     

硬盘:100GB

     

机器类型:n1-standard-4(4个vCPU,15 GB内存)

实施

2个MEMSQL NODES在以下端口上的同一台机器上运行

  

3306 Master Aggregator

     

3307 Leaf

资源利用率

  

内存14.16 GB / 14.69 GB

     

分页0 B / s

     

数据库大小 - 10MB

1818    memsql  1.1%    77% /var/lib/memsql/leaf-3307/memsqld --defaults-file=/var/lib/memsql/leaf-3307/memsql.cnf --pid-file=/var/lib/memsql/leaf-3307/data/memsqld.pid --user=memsql
2736    memsql  0.3%    16% /var/lib/memsql/master-3306/memsqld --defaults-file=/var/lib/memsql/master-330

注意:服务器中没有实现Swap内存。

通过在information_schema.TABLES上运行查询来获取数据库大小。

所有数据都作为行存储存在,因为我们必须通过考虑表之间的许多关系来运行查询。

一旦memsql启动,内存上升到70%并继续增加,2-3小时后memsql在尝试连接时出现以下错误,之后连接也无法完成。

OperationalError: (1836, "Leaf 'xx.xxx.x.xx':3307 failed while executing this query. Try re-running the query.")
[Mon Mar 27 09:26:31.163455 2017] [:error] [pid 1718] [remote xxx.xxx.xxx.xxx:9956]

唯一的解决方案是重新启动服务器,因为它占用了所有内存。

我能为此做些什么?它的实施方式是否存在问题?我应该在这里附上任何日志吗?

Show status extended;查询提供以下结果

+-------------------------------------+------------------------------------------------------------------------+
| Variable_name                       | Value                                                                  |
+-------------------------------------+------------------------------------------------------------------------+
| Aborted_clients                     | 48                                                                     |
| Aborted_connects                    | 1                                                                      |
| Bytes_received                      | 85962135                                                               |
| Bytes_sent                          | 545322701                                                              |
| Connections                         | 1626                                                                   |
| Max_used_connections                | 69                                                                     |
| Queries                             | 364793                                                                 |
| Questions                           | 364793                                                                 |
| Threads_cached                      | 19                                                                     |
| Threads_connected                   | 50                                                                     |
| Threads_created                     | 69                                                                     |
| Threads_running                     | 1                                                                      |
| Threads_background                  | 1                                                                      |
| Threads_idle                        | 0                                                                      |
| Ready_queue                         | 0                                                                      |
| Idle_queue                          | 0                                                                      |
| Context_switches                    | 1626                                                                   |
| Context_switch_misses               | 0                                                                      |
| Uptime                              | 22270                                                                  |
| Auto_attach_remaining_seconds       | 0                                                                      |
| Data_directory                      | /var/lib/memsql/leaf-3307/data                                         |
| Plancache_directory                 | /var/lib/memsql/leaf-3307/plancache                                    |
| Transaction_logs_directory          | /var/lib/memsql/leaf-3307/data/logs                                    |
| Segments_directory                  | /var/lib/memsql/leaf-3307/data/columns                                 |
| Snapshots_directory                 | /var/lib/memsql/leaf-3307/data/snapshots                               |
| Threads_waiting_for_disk_space      | 0                                                                      |
| Seconds_until_expiration            | -1                                                                     |
| License_key                         | 11111111111111111111111111111111                                       |
| License_type                        | community                                                              |
| Query_compilations                  | 62                                                                     |
| Query_compilation_failures          | 0                                                                      |
| GCed_versions_last_sweep            | 0                                                                      |
| Average_garbage_collection_duration | 21 ms                                                                  |
| Total_server_memory                 | 9791.4 MB                                                              |
| Alloc_thread_stacks                 | 70.0 MB                                                                |
| Malloc_active_memory                | 1254.7 (+0.0) MB                                                       |
| Malloc_cumulative_memory            | 7315.5 (+0.2) MB                                                       |
| Buffer_manager_memory               | 1787.8 MB                                                              |
| Buffer_manager_cached_memory        | 77.2 (-0.1) MB                                                         |
| Buffer_manager_unrecycled_memory    | 0.0 MB                                                                 |
| Alloc_skiplist_tower                | 263.8 MB                                                               |
| Alloc_variable                      | 501.4 MB                                                               |
| Alloc_large_variable                | 2.4 MB                                                                 |
| Alloc_table_primary                 | 752.6 MB                                                               |
| Alloc_deleted_version               | 92.9 MB                                                                |
| Alloc_internal_key_node             | 72.1 MB                                                                |
| Alloc_hash_buckets                  | 459.1 MB                                                               |
| Alloc_table_metadata_cache          | 1.1 MB                                                                 |
| Alloc_unit_images                   | 34.8 MB                                                                |
| Alloc_unit_ifn_thunks               | 0.6 MB                                                                 |
| Alloc_object_code_images            | 11.6 MB                                                                |
| Alloc_compiled_unit_sections        | 17.3 MB                                                                |
| Alloc_databases_list_entry          | 17.9 MB                                                                |
| Alloc_plan_cache                    | 0.1 MB                                                                 |
| Alloc_replication_large             | 232.0 MB                                                               |
| Alloc_durability_large              | 7239.1 MB                                                              |
| Alloc_sharding_partitions           | 0.1 MB                                                                 |
| Alloc_security                      | 0.1 MB                                                                 |
| Alloc_log_replay                    | 0.9 MB                                                                 |
| Alloc_client_connection             | 3.0 MB                                                                 |
| Alloc_protocol_packet               | 6.1 (+0.1) MB                                                          |
| Alloc_large_incremental             | 0.8 MB                                                                 |
| Alloc_table_memory                  | 2144.2 MB                                                              |
| Alloc_variable_bucket_16            | allocs:10877846  alloc_MB:166.0  buffer_MB:179.0  cached_buffer_MB:1.9 |
| Alloc_variable_bucket_24            | allocs:4275659  alloc_MB:97.9  buffer_MB:106.8  cached_buffer_MB:1.9   |
| Alloc_variable_bucket_32            | allocs:2875801  alloc_MB:87.8  buffer_MB:93.4  cached_buffer_MB:1.9    |
| Alloc_variable_bucket_40            | allocs:724489  alloc_MB:27.6  buffer_MB:31.0  cached_buffer_MB:1.2     |
| Alloc_variable_bucket_48            | allocs:377060  alloc_MB:17.3  buffer_MB:19.8  cached_buffer_MB:0.9     |
| Alloc_variable_bucket_56            | allocs:228720  alloc_MB:12.2  buffer_MB:14.0  cached_buffer_MB:0.8     |
| Alloc_variable_bucket_64            | allocs:150214  alloc_MB:9.2  buffer_MB:10.1  cached_buffer_MB:0.2      |
| Alloc_variable_bucket_72            | allocs:35264  alloc_MB:2.4  buffer_MB:2.9  cached_buffer_MB:0.0        |
| Alloc_variable_bucket_80            | allocs:14920  alloc_MB:1.1  buffer_MB:1.2  cached_buffer_MB:0.0        |
| Alloc_variable_bucket_88            | allocs:5582  alloc_MB:0.5  buffer_MB:0.6  cached_buffer_MB:0.0         |
| Alloc_variable_bucket_104           | allocs:8075  alloc_MB:0.8  buffer_MB:1.0  cached_buffer_MB:0.0         |
| Alloc_variable_bucket_128           | allocs:8892  alloc_MB:1.1  buffer_MB:1.2  cached_buffer_MB:0.0         |
| Alloc_variable_bucket_160           | allocs:17614  alloc_MB:2.7  buffer_MB:3.0  cached_buffer_MB:0.0        |
| Alloc_variable_bucket_200           | allocs:30454  alloc_MB:5.8  buffer_MB:6.9  cached_buffer_MB:0.6        |
| Alloc_variable_bucket_248           | allocs:4875  alloc_MB:1.2  buffer_MB:1.5  cached_buffer_MB:0.2         |
| Alloc_variable_bucket_312           | allocs:371  alloc_MB:0.1  buffer_MB:0.2  cached_buffer_MB:0.0          |
| Alloc_variable_bucket_384           | allocs:30  alloc_MB:0.0  buffer_MB:0.1  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_480           | allocs:11  alloc_MB:0.0  buffer_MB:0.1  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_600           | allocs:57  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_752           | allocs:62  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_936           | allocs:42  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_1168          | allocs:106  alloc_MB:0.1  buffer_MB:0.2  cached_buffer_MB:0.0          |
| Alloc_variable_bucket_1480          | allocs:126  alloc_MB:0.2  buffer_MB:0.2  cached_buffer_MB:0.0          |
| Alloc_variable_bucket_1832          | allocs:0  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.2            |
| Alloc_variable_bucket_2288          | allocs:1  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.1            |
| Alloc_variable_bucket_2832          | allocs:33  alloc_MB:0.1  buffer_MB:1.1  cached_buffer_MB:0.2           |
| Alloc_variable_bucket_3528          | allocs:16  alloc_MB:0.1  buffer_MB:0.5  cached_buffer_MB:0.1           |
| Alloc_variable_bucket_4504          | allocs:49  alloc_MB:0.2  buffer_MB:0.8  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_5680          | allocs:66  alloc_MB:0.4  buffer_MB:1.2  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_6224          | allocs:30  alloc_MB:0.2  buffer_MB:1.0  cached_buffer_MB:0.1           |
| Alloc_variable_bucket_7264          | allocs:94  alloc_MB:0.7  buffer_MB:1.5  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_9344          | allocs:70  alloc_MB:0.6  buffer_MB:2.6  cached_buffer_MB:0.2           |
| Alloc_variable_bucket_11896         | allocs:14  alloc_MB:0.2  buffer_MB:2.4  cached_buffer_MB:1.2           |
| Alloc_variable_bucket_14544         | allocs:7  alloc_MB:0.1  buffer_MB:2.4  cached_buffer_MB:1.9            |
| Alloc_variable_bucket_18696         | allocs:18  alloc_MB:0.3  buffer_MB:3.2  cached_buffer_MB:1.9           |
| Alloc_variable_bucket_21816         | allocs:4  alloc_MB:0.1  buffer_MB:0.4  cached_buffer_MB:0.0            |
| Alloc_variable_bucket_26184         | allocs:6  alloc_MB:0.1  buffer_MB:0.9  cached_buffer_MB:0.2            |
| Alloc_variable_bucket_32728         | allocs:13  alloc_MB:0.4  buffer_MB:2.4  cached_buffer_MB:1.4           |
| Alloc_variable_bucket_43648         | allocs:12  alloc_MB:0.5  buffer_MB:1.4  cached_buffer_MB:0.2           |
| Alloc_variable_bucket_65472         | allocs:7  alloc_MB:0.4  buffer_MB:2.8  cached_buffer_MB:1.9            |
| Alloc_variable_bucket_130960        | allocs:3  alloc_MB:0.4  buffer_MB:2.2  cached_buffer_MB:1.9            |
| Alloc_variable_cached_buffers       | 21.4 MB                                                                |
| Alloc_variable_allocated            | 438.7 MB                                                               |
| Successful_read_queries             | 9048                                                                   |
| Successful_write_queries            | 19096                                                                  |
| Failed_read_queries                 | 0                                                                      |
| Failed_write_queries                | 4                                                                      |
| Rows_returned_by_reads              | 75939                                                                  |
| Rows_affected_by_writes             | 245                                                                    |
| Execution_time_of_reads             | 7864 ms                                                                |
| Execution_time_of_write             | 180311 ms                                                              |
| Transaction_buffer_wait_time        | 0 ms                                                                   |
| Transaction_log_flush_wait_time     | 0 ms                                                                   |
| Row_lock_wait_time                  | 0 ms                                                                   |
| Ssl_accept_renegotiates             | 0                                                                      |
| Ssl_accepts                         | 0                                                                      |
| Ssl_callback_cache_hits             | 0                                                                      |
| Ssl_client_connects                 | 0                                                                      |
| Ssl_connect_renegotiates            | 0                                                                      |
| Ssl_ctx_verify_depth                | 18446744073709551615                                                   |
| Ssl_ctx_verify_mode                 | 0                                                                      |
| Ssl_default_timeout                 | 0                                                                      |
| Ssl_finished_accepts                | 0                                                                      |
| Ssl_finished_connects               | 0                                                                      |
| Ssl_session_cache_hits              | 0                                                                      |
| Ssl_session_cache_misses            | 0                                                                      |
| Ssl_session_cache_overflows         | 0                                                                      |
| Ssl_session_cache_size              | 20480                                                                  |
| Ssl_session_cache_timeouts          | 0                                                                      |
| Ssl_sessions_reused                 | 0                                                                      |
| Ssl_used_session_cache_entries      | 0                                                                      |
| Ssl_verify_depth                    | 0                                                                      |
| Ssl_verify_mode                     | 0                                                                      |
| Ssl_cipher                          |                                                                        |
| Ssl_cipher_list                     |                                                                        |
| Ssl_version                         |                                                                        |
| Ssl_session_cache_mode              | SERVER                                                                 |
+-------------------------------------+------------------------------------------------------------------------+

1 个答案:

答案 0 :(得分:2)

从状态输出中,我们可以看到:

  • 叶子节点上的总内存为10GB
  • 7GB Alloc_durability_large

您可以在此处查看这些变量的含义:https://help.memsql.com/hc/en-us/articles/115001091386-What-Is-Using-Memory-on-My-Leaves-

最有趣的是Alloc_durability_large中的大量数据,这是不寻常的。你有大量的数据库和/或分区吗? (您可以通过计算叶节点上的SHOW DATABASES EXTENDED中的行数进行检查。)每个都需要一定量的事务缓冲区内存(默认为64 MB)。