我在Ubuntu 16.04.1 LTS上使用5.7.16-10-log Percona Server。
我最近将我的AWS ec2实例从m4.2xlarge实例类型升级到m4.4xlarge。升级后,我的mysql错误日志会定期填写以下行:
2017-09-15T17:10:30.678258+05:30 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4460ms. The settings might not be optimal. (flushed=140, during the time.)
2017-09-15T17:14:31.048443+05:30 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 6047ms. The settings might not be optimal. (flushed=107, during the time.)
2017-09-15T17:15:06.312119+05:30 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 8262ms. The settings might not be optimal. (flushed=64, during the time.)
当Percona Innobackupex按计划运行完整备份和增量备份时,这些都很常见。在这种情况下,这里是根据iostat的磁盘使用情况:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 10.00 0.00 2.00 0.00 48.00 48.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 21.00 0.00 672.00 816.00 63328.00 104448.00 225.51 30.65 20.74 19.28 21.94 0.67 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 4.95 630.69 802.97 62875.25 102677.23 230.95 30.27 21.11 19.80 22.14 0.69 99.01
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 23.00 153.00 988.00 668.00 94888.00 72104.00 201.68 26.84 16.21 14.57 18.62 0.59 98.40
我继续在mysql错误日志中收到这些消息,即使从默认值1024下降lru扫描深度后,我发现在网络上的某些类似帖子上推荐这个消息。
SET GLOBAL innodb_lru_scan_depth=256;
根据某人的推荐,我还试图通过使用以下方式查明问题是否与LRU列表或刷新列表有关:
SHOW ENGINE INNODB STATUS\G
在下面的输出中,在某些情况下,我能够在待处理的写入中看到大于零的刷新列表:
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 50300190720
Dictionary memory allocated 18819582
Internal hash tables (constant factor + variable factor)
Adaptive hash index 3805993024 (764908736 + 3041084288)
Page hash 5976584 (buffer pool 0 only)
Dictionary cache 210046766 (191227184 + 18819582)
File system 1570552 (812272 + 758280)
Lock system 119847848 (119530904 + 316944)
Recovery system 0 (0 + 0)
Buffer pool size 2948760
Buffer pool size, bytes 0
Free buffers 417019
Database pages 2346130
Old database pages 866201
Modified db pages 60127
Pending reads 0
Pending writes: LRU 0, flush list 10, single page 0
Pages made young 5082, not young 0
0.17 youngs/s, 0.00 non-youngs/s
Pages read 2139505, created 206625, written 1680831
1323.32 reads/s, 0.70 creates/s, 21.87 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 1314.62/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2346130, unzip_LRU len: 0
I/O sum[0]:cur[128], unzip sum[0]:cur[0]
这是我使用的my.cnf的片段:
[mysqld]
# GENERAL #
user = mysql
default-storage-engine = InnoDB
socket = /var/run/mysqld/mysqld.sock
pid-file = /var/run/mysqld/mysqld.pid
basedir = /usr
tmpdir = /var/tmp
lc-messages-dir = /usr/share/mysql
explicit_defaults_for_timestamp
log_timestamps = SYSTEM
skip-name-resolve
net-write-timeout = 600
net-read-timeout = 600
innodb-page-cleaners = 8
performance-schema = OFF
# MyISAM #
key-buffer-size = 32M
# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
sysdate-is-now = 1
# DATA STORAGE #
datadir = /var/lib/mysql/
# BINARY LOGGING #
server-id = 2
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
max_binlog_size = 1G
max_binlog_files = 20
# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 50
open-files-limit = 4510
table-definition-cache = 4096
table-open-cache = 4096
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 512M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 45G
innodb-print-all-deadlocks = ON
# LOGGING #
log-error = /var/log/mysql/mysql-error.log
slow-query-log-file = /var/log/mysql/mysql-slow.log
lower-case-table-names = 1
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
#Slow query queries
slow-query-log = 1
long-query-time = 10
slow-query-log-always-write-time = 15 #Slow query queries
log-slow-verbosity = full
log-slow-rate-type = query
log-slow-rate-limit = 100 #queries logged/sec
log-queries-not-using-indexes = 1
任何有关排除故障并解决此问题的建议/帮助/指标都会有很大帮助。
这里有什么限制 - 在我的制作中看起来像:
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 257583
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 257583
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
答案 0 :(得分:1)
您的打开文件限制为1024,将其提升至90000。 RAM中每个文件的操作系统成本几乎为零。 实施后验证您的更改。 然后你的配置部分要做的事情只有1改变,监视 如果没有负面影响,请再次改变下一个工作日。 许多都可以动态更改,请参阅用户指南。
thread_cache_size=100 # from 50 to minimize
threads_created
open_files_limit=30000 # from 4510
table_definition_cache=10000 # from 4096
table_open_cache=10000 # from 4096
这几天值得修改/监控将改善您的实例。
在另一天(或一小时)进行一次不使用索引的查询焦点
slow_query_log=1 # for ON
min_examined_row_limit=1 # to eliminate useless chatter
log_queries_not_using_indexes=1
long_query_time=5000 # for 5000 seconds which will be RARE
避免混淆慢查询和不使用索引信息的查询
完成此单一焦点后,
min_examined_row_limit=0 for useless chatter in the slow query log
log_queries_not_using_indexes=0
long_query_time=10 or less depending on your need to monitor.
进一步分析, 你有多少RAM,你使用SSD还是旋转硬盘? 请发布A)SHOW GLOBAL STATUS的TEXT结果; B)显示全球变量;
答案 1 :(得分:0)
在my.cnf / ini [mysqld]栏目中可以做的事情
将innodb_page_cleaners移至#INNODB#部分并更改为
innodb_page_cleaners=64 # will autolimit to be = innodb_buffer_pool_instances
添加一行
innodb_buffer_pool_instances=8 # from default of 1 to reduce mutex contention
please POST in original question results of
ulimit -a so we can see the Ubuntu limits.