我在GKE本地SSD磁盘上运行带有TiDB的sysbench OLTP基准测试。但是与GKE永久性SSD磁盘相比,我的性能下降了。默认情况下,如何在GKE本地SSD磁盘上获得预期的IOPS性能?
我已经使用psync
引擎运行TiDB OLTP基准测试和fio基准测试,但是结果都显示本地SSD磁盘上的IOPS比持久性SSD磁盘差。而且我还进行了详尽的blktrace分析。我运行的fio命令是:
fio -ioengine=psync -bs=32k -fdatasync=1 -thread -rw=write -size=10G -filename=test -name="max throughput" -iodepth=1 -runtime=60 -numjobs=4 -group_reporting
本地SSD磁盘和永久性磁盘的基准测试结果为:
| disk type | iops | bandwidth |
|---------------------+------+-----------|
| local SSD disk | 302 | 9912kB/s |
| persistent SSD disk | 1149 | 37.7MB/s |
blktrace btt的结果是:
==================== All Devices ====================
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000000002 0.003716416 14.074086987 34636
Q2G 0.000000236 0.000005730 0.005347758 25224
G2I 0.000000727 0.000005446 0.002450425 20575
Q2M 0.000000175 0.000000716 0.000027069 9447
I2D 0.000000778 0.000003197 0.000111657 20538
M2D 0.000001941 0.000011350 0.000431655 9447
D2C 0.000065510 0.000182827 0.001366980 34634
Q2C 0.000072793 0.001181298 0.023394568 34634
==================== Device Overhead ====================
DEV | Q2G G2I Q2M I2D D2C
---------- | --------- --------- --------- --------- ---------
( 8, 48) | 0.3532% 0.2739% 0.0165% 0.1605% 15.4768%
---------- | --------- --------- --------- --------- ---------
Overall | 0.3532% 0.2739% 0.0165% 0.1605% 15.4768%
根据optimization guide,我已经使用nobarrier
选项手动重新安装了磁盘,并且blktrace btt结果看起来很正常。
==================== All Devices ====================
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000000006 0.000785969 12.031454829 123537
Q2G 0.000003929 0.000006162 0.005294881 94553
G2I 0.000004677 0.000029263 0.004555917 94553
Q2M 0.000004069 0.000005337 0.000328930 29019
I2D 0.000005166 0.000020476 0.001078527 94516
M2D 0.000012816 0.000056839 0.001113739 29019
D2C 0.000081435 0.000358712 0.006724447 123535
Q2C 0.000113965 0.000415489 0.006763290 123535
==================== Device Overhead ====================
DEV | Q2G G2I Q2M I2D D2C
---------- | --------- --------- --------- --------- ---------
( 8, 48) | 1.1351% 5.3907% 0.3017% 3.7705% 86.3348%
---------- | --------- --------- --------- --------- ---------
Overall | 1.1351% 5.3907% 0.3017% 3.7705% 86.3348%
但是,根据RedHat’s document,nobarrier
挂载选项仅会对性能产生很小的负面影响(大约3%),不建议在虚拟机上配置的存储上使用它
在Red Hat Enterprise Linux 6中不再建议使用nobarrier,因为写障碍对性能的负面影响可以忽略不计(大约3%)。写屏障的好处通常超过禁用它们的性能好处。另外,绝对不要在虚拟机上配置的存储上使用nobarrier选项。
除了nobarrier
选项外,本地SSD磁盘优化指南还建议安装Linux Guest Environment,但指出它已安装在较新的VM映像上。但是,我发现它没有安装在GKE节点上。
所以我手动安装了Linux Guest Environment,并再次进行了测试,这次的btt结果看起来与预期的一样:
==================== All Devices ====================
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000000001 0.000472816 21.759721028 301371
Q2G 0.000000215 0.000000925 0.000110353 246390
G2I 0.000000279 0.000003579 0.003997348 246390
Q2M 0.000000175 0.000000571 0.000106259 54982
I2D 0.000000609 0.000002635 0.004064992 246390
M2D 0.000001400 0.000005728 0.000509868 54982
D2C 0.000051100 0.000451895 0.009107264 301372
Q2C 0.000054091 0.000458881 0.009111984 301372
==================== Device Overhead ====================
DEV | Q2G G2I Q2M I2D D2C
---------- | --------- --------- --------- --------- ---------
( 8, 80) | 0.1647% 0.6376% 0.0227% 0.4695% 98.4778%
---------- | --------- --------- --------- --------- ---------
Overall | 0.1647% 0.6376% 0.0227% 0.4695% 98.4778%
那么默认情况下如何在GKE本地SSD磁盘上获得预期的IOPS性能而无需进行额外的调整?