Question

也许你可以帮助我，我有点迷失，我试图获得一些额外的IOP和吞吐量将8个本地SSD放在RAID（mdadm）上并且我无法超越平均性能一个本地的ssd驱动器。我正在使用fio来运行测试，到目前为止最好的raid配置是RAID 0，它显示出与单个驱动器几乎相同的性能。

我也尝试过RAID 5，由于RAID损失，看起来性能下降（特别是在写入时）。所以看起来实例上有一个IO限制，无论我放在它上面的本地SSD的数量，我得到相同的统计数据，如果RAID配置有一个惩罚我得到更低的统计数据任何想法？是否存在实例的IO限制（对于测试，我在内存中使用32位虚拟机CentOS / Debian和208Gb。

一些输出：

RAID 0：

root@sgs02 data]# /usr/local/bin/fio --ioengine=libaio --direct=1
--name=test --filename=/data/t22 --bs=4k --iodep th=128 --size=4G --randrepeat=1 --readwrite=randrw --rwmixread=70 test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128 fio-2.13 Starting 1 process Jobs: 1 (f=1): [m(1)] [100.0% done] [313.8MB/132.6MB/0KB /s] [80.4K/33.1K/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=13748: Mon Aug 29 07:19:10 2016   read : io=2865.9MB, bw=323120KB/s, iops=80780, runt=  9082msec
    slat (usec): min=2, max=167, avg= 7.22, stdev= 4.42
    clat (usec): min=125, max=9876, avg=1128.41, stdev=298.77
     lat (usec): min=132, max=9881, avg=1135.63, stdev=298.82
    clat percentiles (usec):
     |  1.00th=[  908],  5.00th=[  956], 10.00th=[  980], 20.00th=[ 1012],
     | 30.00th=[ 1032], 40.00th=[ 1064], 50.00th=[ 1080], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1160], 90.00th=[ 1208], 95.00th=[ 1288],
     | 99.00th=[ 2928], 99.50th=[ 3472], 99.90th=[ 3920], 99.95th=[ 4320],
     | 99.99th=[ 5088]   write: io=1230.3MB, bw=138706KB/s, iops=34676, runt=  9082msec
    slat (usec): min=2, max=116, avg= 8.05, stdev= 4.64
    clat (usec): min=87, max=8943, avg=1034.94, stdev=146.07
     lat (usec): min=94, max=8947, avg=1042.99, stdev=146.39
    clat percentiles (usec):
     |  1.00th=[  852],  5.00th=[  900], 10.00th=[  924], 20.00th=[  964],
     | 30.00th=[  988], 40.00th=[ 1004], 50.00th=[ 1032], 60.00th=[ 1048],
     | 70.00th=[ 1064], 80.00th=[ 1096], 90.00th=[ 1144], 95.00th=[ 1176],
     | 99.00th=[ 1272], 99.50th=[ 1320], 99.90th=[ 3184], 99.95th=[ 3984],
     | 99.99th=[ 5856]
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=21.69%
    lat (msec) : 2=76.85%, 4=1.35%, 10=0.07%

RAID 5：

/usr/local/bin/fio --ioengine=libaio --direct=1 --name=test
--filename=/data/t22 --bs=4k --iodepth=128 --size=4G --randrepeat=1 --readwrite=ra ndrw --rwmixread=70 test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128 fio-2.13 Starting 1 process Jobs: 1 (f=1): [m(1)] [100.0% done] [248.7MB/104.6MB/0KB /s] [63.7K/26.8K/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=10743: Mon Aug 29 07:19:10 2016   read : io=2865.9MB, bw=235218KB/s, iops=58804, runt= 12476msec
    slat (usec): min=2, max=402, avg= 9.43, stdev= 8.99
    clat (usec): min=100, max=15268, avg=1155.38, stdev=790.57
     lat (usec): min=108, max=15271, avg=1164.81, stdev=792.64
    clat percentiles (usec):
     |  1.00th=[  119],  5.00th=[  137], 10.00th=[  155], 20.00th=[  195],
     | 30.00th=[  286], 40.00th=[ 1320], 50.00th=[ 1400], 60.00th=[ 1464],
     | 70.00th=[ 1512], 80.00th=[ 1624], 90.00th=[ 1928], 95.00th=[ 2096],
     | 99.00th=[ 3440], 99.50th=[ 3952], 99.90th=[ 5600], 99.95th=[ 6432],
     | 99.99th=[14656]   write: io=1230.3MB, bw=100972KB/s, iops=25243, runt= 12476msec
    slat (usec): min=2, max=13853, avg= 9.33, stdev=29.91
    clat (usec): min=212, max=424477, avg=2341.19, stdev=1500.44
     lat (usec): min=217, max=424481, avg=2350.52, stdev=1499.50
    clat percentiles (usec):
     |  1.00th=[ 1336],  5.00th=[ 1448], 10.00th=[ 1512], 20.00th=[ 1592],
     | 30.00th=[ 1656], 40.00th=[ 1752], 50.00th=[ 2024], 60.00th=[ 2288],
     | 70.00th=[ 2704], 80.00th=[ 3056], 90.00th=[ 3568], 95.00th=[ 3984],
     | 99.00th=[ 5280], 99.50th=[ 6048], 99.90th=[ 8256], 99.95th=[15168],
     | 99.99th=[20352]
    lat (usec) : 250=19.32%, 500=3.46%, 750=0.40%, 1000=0.34%
    lat (msec) : 2=56.10%, 4=18.56%, 10=1.79%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%   cpu          : usr=7.26%, sys=87.08%, ctx=43414, majf=0, minf=33

单驱动

/usr/local/bin/fio --ioengine=libaio --direct=1 --name=test
--filename=/data/t22 --bs=4k --iode pth=128 --size=4G --randrepeat=1 --readwrite=randrw --rwmixread=70 test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128 fio-2.13 Starting 1 process Jobs: 1 (f=1): [m(1)] [100.0% done] [328.1MB/138.5MB/0KB /s] [83.1K/35.5K/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=4554: Mon Aug 29 07:19:19 2016   read : io=2865.9MB, bw=338163KB/s, iops=84540, runt=  8678msec
    slat (usec): min=1, max=199, avg= 3.41, stdev= 4.70
    clat (usec): min=98, max=19530, avg=1226.75, stdev=1142.24
     lat (usec): min=103, max=19532, avg=1230.16, stdev=1142.32
    clat percentiles (usec):
     |  1.00th=[  153],  5.00th=[  243], 10.00th=[  302], 20.00th=[  386],
     | 30.00th=[  458], 40.00th=[  548], 50.00th=[  676], 60.00th=[  980],
     | 70.00th=[ 1480], 80.00th=[ 2160], 90.00th=[ 2928], 95.00th=[ 3600],
     | 99.00th=[ 4768], 99.50th=[ 5216], 99.90th=[ 5856], 99.95th=[ 6688],
     | 99.99th=[15936]   write: io=1230.3MB, bw=145163KB/s, iops=36290, runt=  8678msec
    slat (usec): min=1, max=196, avg= 4.14, stdev= 5.17
    clat (usec): min=29, max=16473, avg=654.72, stdev=732.68
     lat (usec): min=45, max=16476, avg=658.86, stdev=732.68
    clat percentiles (usec):
     |  1.00th=[   59],  5.00th=[   88], 10.00th=[  123], 20.00th=[  193],

Answer 1

你可能在mdadm遇到了瓶颈。查看performance of raid5 on fast devices线程，该线程讨论了针对某些操作使用单个内核线程的问题。虽然这谈到了RAID 5，但是很多代码都被重用了，所以它可能是RAID 0中的类似问题。他们正在努力解决这个瓶颈。

本地ssd Raid mdadm表现

1 个答案: