== sanity test 398b: DIO and buffer IO race ============== 09:32:44 (1713447164) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.16889 s, 43.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17651: Thu Apr 18 09:32:59 2024 read: IOPS=127, BW=508KiB/s (521kB/s)(5992KiB/11785msec) clat (usec): min=50, max=158557, avg=4504.45, stdev=4680.60 lat (usec): min=50, max=158558, avg=4505.32, stdev=4680.78 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 147], 10.00th=[ 2114], 20.00th=[ 2573], | 30.00th=[ 2999], 40.00th=[ 3490], 50.00th=[ 4015], 60.00th=[ 4621], | 70.00th=[ 5211], 80.00th=[ 5997], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[ 13566], 99.50th=[ 15664], 99.90th=[ 20579], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 280, max= 584, per=25.24%, avg=502.48, stdev=72.81, samples=23 iops : min= 70, max= 146, avg=125.57, stdev=18.23, samples=23 write: IOPS=133, BW=534KiB/s (547kB/s)(6296KiB/11785msec) clat (usec): min=398, max=19952, avg=3162.04, stdev=2880.05 lat (usec): min=398, max=19952, avg=3162.72, stdev=2880.07 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 742], 40.00th=[ 1680], 50.00th=[ 2573], 60.00th=[ 3130], | 70.00th=[ 3949], 80.00th=[ 5211], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[12911], 99.50th=[15008], 99.90th=[16712], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 352, max= 672, per=25.35%, avg=528.52, stdev=75.13, samples=23 iops : min= 88, max= 168, avg=132.09, stdev=18.75, samples=23 lat (usec) : 100=2.28%, 250=0.20%, 500=1.24%, 750=14.45%, 1000=2.25% lat (msec) : 2=4.79%, 4=35.06%, 10=36.91%, 20=2.77%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=23.12%, ctx=5942, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17652: Thu Apr 18 09:32:59 2024 read: IOPS=125, BW=502KiB/s (515kB/s)(5940KiB/11821msec) clat (usec): min=47, max=24534, avg=4354.20, stdev=2525.38 lat (usec): min=47, max=24534, avg=4354.77, stdev=2525.39 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 233], 10.00th=[ 2114], 20.00th=[ 2474], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 6128], 90.00th=[ 7177], 95.00th=[ 8717], | 99.00th=[12780], 99.50th=[13960], 99.90th=[20841], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 272, max= 624, per=24.96%, avg=496.96, stdev=73.36, samples=23 iops : min= 68, max= 156, avg=124.17, stdev=18.36, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6348KiB/11821msec) clat (usec): min=406, max=154980, avg=3340.70, stdev=4747.99 lat (usec): min=407, max=154981, avg=3342.57, stdev=4747.50 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 816], 40.00th=[ 2040], 50.00th=[ 2671], 60.00th=[ 3195], | 70.00th=[ 4146], 80.00th=[ 5473], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[ 12125], 99.50th=[ 13042], 99.90th=[ 22938], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 344, max= 688, per=25.10%, avg=523.35, stdev=82.45, samples=23 iops : min= 86, max= 172, avg=130.78, stdev=20.63, samples=23 lat (usec) : 50=0.16%, 100=2.15%, 250=0.13%, 500=1.20%, 750=13.48% lat (usec) : 1000=2.60% lat (msec) : 2=4.43%, 4=36.26%, 10=36.85%, 20=2.54%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.12%, ctx=6184, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17653: Thu Apr 18 09:32:59 2024 read: IOPS=125, BW=500KiB/s (512kB/s)(6028KiB/12055msec) clat (usec): min=47, max=22464, avg=4439.37, stdev=2370.50 lat (usec): min=48, max=22464, avg=4440.22, stdev=2370.48 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1729], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 2966], 40.00th=[ 3490], 50.00th=[ 4146], 60.00th=[ 4686], | 70.00th=[ 5276], 80.00th=[ 6063], 90.00th=[ 7308], 95.00th=[ 8586], | 99.00th=[12256], 99.50th=[13435], 99.90th=[15401], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 168, max= 608, per=24.49%, avg=487.57, stdev=87.82, samples=23 iops : min= 42, max= 152, avg=121.83, stdev=21.98, samples=23 write: IOPS=129, BW=519KiB/s (532kB/s)(6260KiB/12055msec) clat (usec): min=402, max=170693, avg=3400.12, stdev=5077.04 lat (usec): min=403, max=170694, avg=3400.80, stdev=5077.06 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 799], 40.00th=[ 2114], 50.00th=[ 2769], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5604], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[ 11469], 99.50th=[ 13698], 99.90th=[ 19268], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 216, max= 648, per=24.28%, avg=506.30, stdev=89.53, samples=23 iops : min= 54, max= 162, avg=126.52, stdev=22.36, samples=23 lat (usec) : 50=0.10%, 100=1.95%, 250=0.20%, 500=1.73%, 750=13.02% lat (usec) : 1000=2.08% lat (msec) : 2=4.36%, 4=34.73%, 10=39.06%, 20=2.70%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.22%, ctx=6073, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17654: Thu Apr 18 09:32:59 2024 read: IOPS=127, BW=512KiB/s (524kB/s)(6052KiB/11827msec) clat (usec): min=47, max=20364, avg=4352.94, stdev=2463.23 lat (usec): min=48, max=20365, avg=4354.35, stdev=2462.91 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1696], 10.00th=[ 2180], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3359], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 7242], 95.00th=[ 8356], | 99.00th=[13829], 99.50th=[15008], 99.90th=[19006], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 256, max= 656, per=25.20%, avg=501.74, stdev=73.20, samples=23 iops : min= 64, max= 164, avg=125.39, stdev=18.30, samples=23 write: IOPS=131, BW=527KiB/s (540kB/s)(6236KiB/11827msec) clat (usec): min=402, max=172715, avg=3338.12, stdev=5202.22 lat (usec): min=403, max=172717, avg=3338.95, stdev=5202.27 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 578], | 30.00th=[ 701], 40.00th=[ 1844], 50.00th=[ 2671], 60.00th=[ 3294], | 70.00th=[ 4113], 80.00th=[ 5538], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[ 12387], 99.50th=[ 16909], 99.90th=[ 24511], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 336, max= 800, per=24.93%, avg=519.87, stdev=95.45, samples=23 iops : min= 84, max= 200, avg=129.91, stdev=23.88, samples=23 lat (usec) : 50=0.07%, 100=1.99%, 250=0.23%, 500=1.43%, 750=14.29% lat (usec) : 1000=2.12% lat (msec) : 2=3.81%, 4=36.46%, 10=36.91%, 20=2.60%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.03%, ctx=6100, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1992KiB/s (2040kB/s), 500KiB/s-512KiB/s (512kB/s-524kB/s), io=23.4MiB (24.6MB), run=11785-12055msec WRITE: bw=2085KiB/s (2135kB/s), 519KiB/s-537KiB/s (532kB/s-550kB/s), io=24.6MiB (25.7MB), run=11785-12055msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17646: Thu Apr 18 09:33:02 2024 read: IOPS=96, BW=387KiB/s (396kB/s)(5992KiB/15497msec) clat (usec): min=1409, max=160127, avg=4234.45, stdev=4879.23 lat (usec): min=1410, max=160128, avg=4235.10, stdev=4879.22 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1893], 20.00th=[ 2057], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3097], 60.00th=[ 3720], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 8291], 95.00th=[ 10290], | 99.00th=[ 14222], 99.50th=[ 15664], 99.90th=[ 19792], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 144, max= 840, per=24.99%, avg=382.40, stdev=150.74, samples=30 iops : min= 36, max= 210, avg=95.60, stdev=37.69, samples=30 write: IOPS=101, BW=406KiB/s (416kB/s)(6296KiB/15497msec) clat (usec): min=2215, max=25406, avg=5765.68, stdev=3353.75 lat (usec): min=2216, max=25407, avg=5766.40, stdev=3353.73 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2737], 10.00th=[ 2933], 20.00th=[ 3195], | 30.00th=[ 3556], 40.00th=[ 3982], 50.00th=[ 4621], 60.00th=[ 5407], | 70.00th=[ 6587], 80.00th=[ 7898], 90.00th=[10290], 95.00th=[11994], | 99.00th=[18482], 99.50th=[20841], 99.90th=[24511], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 160, max= 744, per=24.97%, avg=400.00, stdev=145.42, samples=30 iops : min= 40, max= 186, avg=100.00, stdev=36.35, samples=30 lat (msec) : 2=7.91%, 4=44.21%, 10=39.62%, 20=7.91%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.09%, ctx=3609, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17647: Thu Apr 18 09:33:02 2024 read: IOPS=94, BW=379KiB/s (388kB/s)(5940KiB/15684msec) clat (usec): min=1477, max=159551, avg=4309.83, stdev=5045.95 lat (usec): min=1477, max=159552, avg=4310.43, stdev=5045.98 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1893], 20.00th=[ 2057], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3589], | 70.00th=[ 4424], 80.00th=[ 5800], 90.00th=[ 8586], 95.00th=[ 10683], | 99.00th=[ 15401], 99.50th=[ 17171], 99.90th=[ 22414], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 119, max= 752, per=24.69%, avg=377.71, stdev=159.92, samples=31 iops : min= 29, max= 188, avg=94.35, stdev=40.04, samples=31 write: IOPS=101, BW=405KiB/s (414kB/s)(6348KiB/15684msec) clat (usec): min=2210, max=27393, avg=5800.09, stdev=3407.46 lat (usec): min=2210, max=27394, avg=5800.80, stdev=3407.47 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2704], 10.00th=[ 2900], 20.00th=[ 3195], | 30.00th=[ 3490], 40.00th=[ 4047], 50.00th=[ 4621], 60.00th=[ 5407], | 70.00th=[ 6521], 80.00th=[ 8160], 90.00th=[10290], 95.00th=[12125], | 99.00th=[18744], 99.50th=[20579], 99.90th=[27132], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 199, max= 792, per=25.01%, avg=400.68, stdev=144.15, samples=31 iops : min= 49, max= 198, avg=100.10, stdev=36.09, samples=31 lat (msec) : 2=8.24%, 4=43.46%, 10=39.36%, 20=8.46%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.09%, sys=11.00%, ctx=3584, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17649: Thu Apr 18 09:33:02 2024 read: IOPS=97, BW=389KiB/s (398kB/s)(6028KiB/15508msec) clat (usec): min=1475, max=22964, avg=4262.93, stdev=3154.51 lat (usec): min=1475, max=22965, avg=4263.71, stdev=3154.55 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1811], 10.00th=[ 1909], 20.00th=[ 2073], | 30.00th=[ 2311], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3589], | 70.00th=[ 4228], 80.00th=[ 5866], 90.00th=[ 8848], 95.00th=[11207], | 99.00th=[15533], 99.50th=[17695], 99.90th=[22414], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 160, max= 776, per=25.41%, avg=388.77, stdev=144.46, samples=31 iops : min= 40, max= 194, avg=97.13, stdev=36.13, samples=31 write: IOPS=100, BW=404KiB/s (413kB/s)(6260KiB/15508msec) clat (msec): min=2, max=167, avg= 5.75, stdev= 5.16 lat (msec): min=2, max=167, avg= 5.75, stdev= 5.16 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 17], 99.50th=[ 19], 99.90th=[ 27], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 176, max= 792, per=25.17%, avg=403.23, stdev=151.25, samples=31 iops : min= 44, max= 198, avg=100.74, stdev=37.80, samples=31 lat (msec) : 2=7.81%, 4=44.95%, 10=38.96%, 20=7.94%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.10%, ctx=3590, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17650: Thu Apr 18 09:33:02 2024 read: IOPS=97, BW=389KiB/s (398kB/s)(6052KiB/15567msec) clat (usec): min=1531, max=23049, avg=4160.73, stdev=3009.05 lat (usec): min=1531, max=23050, avg=4161.34, stdev=3009.03 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1893], 20.00th=[ 2057], | 30.00th=[ 2278], 40.00th=[ 2638], 50.00th=[ 3097], 60.00th=[ 3589], | 70.00th=[ 4359], 80.00th=[ 5669], 90.00th=[ 8455], 95.00th=[10421], | 99.00th=[15533], 99.50th=[18482], 99.90th=[22414], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 136, max= 856, per=25.32%, avg=387.35, stdev=153.85, samples=31 iops : min= 34, max= 214, avg=96.84, stdev=38.46, samples=31 write: IOPS=100, BW=401KiB/s (410kB/s)(6236KiB/15567msec) clat (msec): min=2, max=168, avg= 5.90, stdev= 5.28 lat (msec): min=2, max=168, avg= 5.90, stdev= 5.28 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 22], 99.90th=[ 28], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 160, max= 736, per=24.92%, avg=399.23, stdev=145.57, samples=31 iops : min= 40, max= 184, avg=99.81, stdev=36.39, samples=31 lat (msec) : 2=8.27%, 4=44.34%, 10=38.96%, 20=7.98%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=10.83%, ctx=3646, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1531KiB/s (1568kB/s), 379KiB/s-389KiB/s (388kB/s-398kB/s), io=23.4MiB (24.6MB), run=15497-15684msec WRITE: bw=1603KiB/s (1641kB/s), 401KiB/s-406KiB/s (410kB/s-416kB/s), io=24.6MiB (25.7MB), run=15497-15684msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17685: Thu Apr 18 09:33:09 2024 read: IOPS=65, BW=1047KiB/s (1072kB/s)(5920KiB/5653msec) clat (usec): min=60, max=39472, avg=10386.27, stdev=3592.87 lat (usec): min=61, max=39473, avg=10386.83, stdev=3592.89 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 6783], 10.00th=[ 7570], 20.00th=[ 8455], | 30.00th=[ 8979], 40.00th=[ 9634], 50.00th=[10028], 60.00th=[10683], | 70.00th=[11338], 80.00th=[12387], 90.00th=[13698], 95.00th=[15008], | 99.00th=[21627], 99.50th=[25035], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 896, max= 1248, per=24.88%, avg=1035.64, stdev=135.21, samples=11 iops : min= 56, max= 78, avg=64.73, stdev= 8.45, samples=11 write: IOPS=70, BW=1126KiB/s (1154kB/s)(6368KiB/5653msec) clat (usec): min=702, max=36040, avg=4532.40, stdev=4692.87 lat (usec): min=702, max=36041, avg=4533.33, stdev=4692.88 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 775], 10.00th=[ 799], 20.00th=[ 930], | 30.00th=[ 2040], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6390], 90.00th=[10290], 95.00th=[14091], | 99.00th=[22414], 99.50th=[34341], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 800, max= 1472, per=25.71%, avg=1125.82, stdev=177.49, samples=11 iops : min= 50, max= 92, avg=70.36, stdev=11.09, samples=11 lat (usec) : 100=0.91%, 250=0.39%, 750=0.65%, 1000=10.42% lat (msec) : 2=4.04%, 4=16.02%, 10=37.63%, 20=28.39%, 50=1.56% cpu : usr=0.02%, sys=19.37%, ctx=2686, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17686: Thu Apr 18 09:33:09 2024 read: IOPS=63, BW=1021KiB/s (1046kB/s)(5744KiB/5624msec) clat (usec): min=73, max=31979, avg=10487.64, stdev=3261.62 lat (usec): min=73, max=31979, avg=10488.20, stdev=3261.61 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 6849], 10.00th=[ 7635], 20.00th=[ 8291], | 30.00th=[ 9110], 40.00th=[ 9634], 50.00th=[10028], 60.00th=[10552], | 70.00th=[11338], 80.00th=[12256], 90.00th=[13960], 95.00th=[15795], | 99.00th=[21890], 99.50th=[26870], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 736, max= 1248, per=24.60%, avg=1024.00, stdev=133.48, samples=11 iops : min= 46, max= 78, avg=64.00, stdev= 8.34, samples=11 write: IOPS=72, BW=1164KiB/s (1192kB/s)(6544KiB/5624msec) clat (usec): min=699, max=57244, avg=4527.44, stdev=4983.34 lat (usec): min=700, max=57247, avg=4528.52, stdev=4983.41 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 775], 10.00th=[ 791], 20.00th=[ 914], | 30.00th=[ 2057], 40.00th=[ 2442], 50.00th=[ 3064], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6652], 90.00th=[ 9634], 95.00th=[13042], | 99.00th=[21627], 99.50th=[26346], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 896, max= 1568, per=26.37%, avg=1154.91, stdev=217.72, samples=11 iops : min= 56, max= 98, avg=72.18, stdev=13.61, samples=11 lat (usec) : 100=0.65%, 250=0.13%, 750=1.30%, 1000=10.29% lat (msec) : 2=3.65%, 4=16.28%, 10=38.15%, 20=27.86%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=18.87%, ctx=2704, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17687: Thu Apr 18 09:33:09 2024 read: IOPS=67, BW=1087KiB/s (1113kB/s)(6256KiB/5754msec) clat (usec): min=61, max=25816, avg=10257.22, stdev=2963.15 lat (usec): min=62, max=25817, avg=10257.82, stdev=2963.14 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 6915], 10.00th=[ 7570], 20.00th=[ 8291], | 30.00th=[ 8979], 40.00th=[ 9503], 50.00th=[10028], 60.00th=[10683], | 70.00th=[11338], 80.00th=[11994], 90.00th=[13304], 95.00th=[15008], | 99.00th=[20579], 99.50th=[23462], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 736, max= 1376, per=26.07%, avg=1085.09, stdev=215.36, samples=11 iops : min= 46, max= 86, avg=67.82, stdev=13.46, samples=11 write: IOPS=65, BW=1048KiB/s (1073kB/s)(6032KiB/5754msec) clat (usec): min=676, max=55182, avg=4607.61, stdev=5359.77 lat (usec): min=677, max=55184, avg=4609.59, stdev=5359.95 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 1680], | 30.00th=[ 2212], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3884], | 70.00th=[ 4752], 80.00th=[ 5932], 90.00th=[ 9634], 95.00th=[13829], | 99.00th=[25035], 99.50th=[39584], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 672, max= 1408, per=23.25%, avg=1018.18, stdev=204.81, samples=11 iops : min= 42, max= 88, avg=63.64, stdev=12.80, samples=11 lat (usec) : 100=0.91%, 250=0.13%, 750=0.65%, 1000=8.20% lat (msec) : 2=2.34%, 4=19.14%, 10=38.67%, 20=28.52%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.00%, sys=18.34%, ctx=2752, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17688: Thu Apr 18 09:33:09 2024 read: IOPS=65, BW=1051KiB/s (1076kB/s)(6032KiB/5739msec) clat (usec): min=61, max=29036, avg=10514.28, stdev=3313.04 lat (usec): min=62, max=29036, avg=10514.84, stdev=3313.05 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 6915], 10.00th=[ 7373], 20.00th=[ 8291], | 30.00th=[ 8979], 40.00th=[ 9765], 50.00th=[10159], 60.00th=[10683], | 70.00th=[11469], 80.00th=[12387], 90.00th=[14091], 95.00th=[15664], | 99.00th=[24249], 99.50th=[27657], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 736, max= 1152, per=24.74%, avg=1029.82, stdev=122.96, samples=11 iops : min= 46, max= 72, avg=64.36, stdev= 7.68, samples=11 write: IOPS=68, BW=1090KiB/s (1116kB/s)(6256KiB/5739msec) clat (usec): min=702, max=52122, avg=4522.42, stdev=5040.52 lat (usec): min=702, max=52124, avg=4523.47, stdev=5040.49 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 775], 10.00th=[ 816], 20.00th=[ 1139], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3589], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[10028], 95.00th=[13304], | 99.00th=[25035], 99.50th=[37487], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 800, max= 1472, per=24.51%, avg=1073.45, stdev=214.84, samples=11 iops : min= 50, max= 92, avg=67.09, stdev=13.43, samples=11 lat (usec) : 100=0.52%, 250=0.26%, 750=0.52%, 1000=8.72% lat (msec) : 2=3.91%, 4=18.88%, 10=35.55%, 20=30.21%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=18.79%, ctx=2713, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4163KiB/s (4263kB/s), 1021KiB/s-1087KiB/s (1046kB/s-1113kB/s), io=23.4MiB (24.5MB), run=5624-5754msec WRITE: bw=4380KiB/s (4485kB/s), 1048KiB/s-1164KiB/s (1073kB/s-1192kB/s), io=24.6MiB (25.8MB), run=5624-5754msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17690: Thu Apr 18 09:33:09 2024 read: IOPS=65, BW=1049KiB/s (1074kB/s)(5920KiB/5644msec) clat (usec): min=1769, max=41039, avg=6075.30, stdev=4878.61 lat (usec): min=1770, max=41040, avg=6075.86, stdev=4878.62 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 3130], 50.00th=[ 3785], 60.00th=[ 5276], | 70.00th=[ 7963], 80.00th=[ 9241], 90.00th=[12125], 95.00th=[15533], | 99.00th=[23462], 99.50th=[24511], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 864, max= 1344, per=25.03%, avg=1038.55, stdev=151.70, samples=11 iops : min= 54, max= 84, avg=64.91, stdev= 9.48, samples=11 write: IOPS=70, BW=1128KiB/s (1155kB/s)(6368KiB/5644msec) clat (usec): min=2567, max=43783, avg=8514.25, stdev=5160.95 lat (usec): min=2569, max=43785, avg=8515.44, stdev=5160.96 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3228], 10.00th=[ 3523], 20.00th=[ 4228], | 30.00th=[ 5211], 40.00th=[ 6194], 50.00th=[ 7308], 60.00th=[ 8848], | 70.00th=[10028], 80.00th=[11994], 90.00th=[14615], 95.00th=[17695], | 99.00th=[24511], 99.50th=[35390], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 768, max= 1408, per=25.79%, avg=1125.82, stdev=181.48, samples=11 iops : min= 48, max= 88, avg=70.36, stdev=11.34, samples=11 lat (msec) : 2=1.17%, 4=34.24%, 10=41.15%, 20=20.70%, 50=2.73% cpu : usr=0.07%, sys=9.71%, ctx=1140, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17691: Thu Apr 18 09:33:09 2024 read: IOPS=63, BW=1022KiB/s (1046kB/s)(5744KiB/5623msec) clat (usec): min=1818, max=37531, avg=5675.42, stdev=4238.80 lat (usec): min=1818, max=37531, avg=5675.99, stdev=4238.81 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3982], 60.00th=[ 5211], | 70.00th=[ 6849], 80.00th=[ 8586], 90.00th=[10814], 95.00th=[14091], | 99.00th=[18220], 99.50th=[30016], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 736, max= 1248, per=24.60%, avg=1020.80, stdev=152.69, samples=10 iops : min= 46, max= 78, avg=63.80, stdev= 9.54, samples=10 write: IOPS=72, BW=1164KiB/s (1192kB/s)(6544KiB/5623msec) clat (usec): min=2648, max=44109, avg=8750.18, stdev=5535.14 lat (usec): min=2649, max=44110, avg=8751.37, stdev=5535.15 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3261], 10.00th=[ 3589], 20.00th=[ 4178], | 30.00th=[ 5473], 40.00th=[ 6259], 50.00th=[ 7373], 60.00th=[ 8717], | 70.00th=[10028], 80.00th=[11731], 90.00th=[15926], 95.00th=[18744], | 99.00th=[31589], 99.50th=[35914], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 928, max= 1568, per=26.32%, avg=1148.80, stdev=209.27, samples=10 iops : min= 58, max= 98, avg=71.80, stdev=13.08, samples=10 lat (msec) : 2=1.30%, 4=31.38%, 10=44.92%, 20=20.70%, 50=1.69% cpu : usr=0.14%, sys=9.87%, ctx=1173, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17692: Thu Apr 18 09:33:09 2024 read: IOPS=67, BW=1084KiB/s (1110kB/s)(6256KiB/5772msec) clat (usec): min=1733, max=37333, avg=6293.11, stdev=4654.09 lat (usec): min=1733, max=37333, avg=6293.68, stdev=4654.10 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2540], | 30.00th=[ 2868], 40.00th=[ 3458], 50.00th=[ 4555], 60.00th=[ 6259], | 70.00th=[ 8291], 80.00th=[10028], 90.00th=[11994], 95.00th=[14484], | 99.00th=[24511], 99.50th=[25297], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 736, max= 1568, per=26.21%, avg=1087.45, stdev=218.32, samples=11 iops : min= 46, max= 98, avg=67.82, stdev=13.61, samples=11 write: IOPS=65, BW=1045KiB/s (1070kB/s)(6032KiB/5772msec) clat (usec): min=2513, max=50421, avg=8763.50, stdev=6092.78 lat (usec): min=2513, max=50422, avg=8764.70, stdev=6092.76 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3261], 10.00th=[ 3556], 20.00th=[ 4015], | 30.00th=[ 5342], 40.00th=[ 6194], 50.00th=[ 7177], 60.00th=[ 8291], | 70.00th=[ 9634], 80.00th=[11338], 90.00th=[17171], 95.00th=[20317], | 99.00th=[38011], 99.50th=[44827], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 768, max= 1280, per=23.32%, avg=1017.73, stdev=132.55, samples=11 iops : min= 48, max= 80, avg=63.45, stdev= 8.31, samples=11 lat (msec) : 2=1.43%, 4=31.51%, 10=43.10%, 20=20.57%, 50=3.26% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=9.74%, ctx=1182, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17693: Thu Apr 18 09:33:09 2024 read: IOPS=68, BW=1098KiB/s (1124kB/s)(6032KiB/5495msec) clat (usec): min=1822, max=44217, avg=5672.76, stdev=4882.01 lat (usec): min=1822, max=44217, avg=5673.32, stdev=4882.01 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2671], 40.00th=[ 2999], 50.00th=[ 3654], 60.00th=[ 4752], | 70.00th=[ 6652], 80.00th=[ 8848], 90.00th=[11338], 95.00th=[13698], | 99.00th=[24511], 99.50th=[38536], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 800, max= 1408, per=25.45%, avg=1056.00, stdev=173.31, samples=10 iops : min= 50, max= 88, avg=66.00, stdev=10.83, samples=10 write: IOPS=71, BW=1138KiB/s (1166kB/s)(6256KiB/5495msec) clat (usec): min=2529, max=33385, avg=8564.33, stdev=5373.42 lat (usec): min=2530, max=33386, avg=8565.54, stdev=5373.44 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3228], 10.00th=[ 3523], 20.00th=[ 3916], | 30.00th=[ 4883], 40.00th=[ 5997], 50.00th=[ 7046], 60.00th=[ 8455], | 70.00th=[ 9896], 80.00th=[12387], 90.00th=[16450], 95.00th=[19268], | 99.00th=[27657], 99.50th=[30016], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 768, max= 1408, per=25.22%, avg=1100.80, stdev=219.23, samples=10 iops : min= 48, max= 88, avg=68.80, stdev=13.70, samples=10 lat (msec) : 2=1.43%, 4=36.46%, 10=39.71%, 20=19.40%, 50=2.99% cpu : usr=0.07%, sys=9.85%, ctx=1128, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4150KiB/s (4249kB/s), 1022KiB/s-1098KiB/s (1046kB/s-1124kB/s), io=23.4MiB (24.5MB), run=5495-5772msec WRITE: bw=4366KiB/s (4471kB/s), 1045KiB/s-1164KiB/s (1070kB/s-1192kB/s), io=24.6MiB (25.8MB), run=5495-5772msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17703: Thu Apr 18 09:33:15 2024 read: IOPS=1, BW=1473KiB/s (1508kB/s)(7168KiB/4867msec) clat (usec): min=417, max=941528, avg=576487.74, stdev=394860.51 lat (usec): min=418, max=941530, avg=576488.79, stdev=394860.64 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 41681], | 30.00th=[583009], 40.00th=[583009], 50.00th=[792724], 60.00th=[801113], | 70.00th=[801113], 80.00th=[868221], 90.00th=[943719], 95.00th=[943719], | 99.00th=[943719], 99.50th=[943719], 99.90th=[943719], 99.95th=[943719], | 99.99th=[943719] bw ( KiB/s): min= 1729, max= 2048, per=44.91%, avg=1889.50, stdev=173.63, samples=6 iops : min= 1, max= 2, avg= 1.50, stdev= 0.55, samples=6 write: IOPS=1, BW=1052KiB/s (1077kB/s)(5120KiB/4867msec) clat (msec): min=26, max=448, avg=166.05, stdev=173.10 lat (msec): min=26, max=448, avg=166.12, stdev=173.10 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 1729, max= 3465, per=36.73%, avg=2163.75, stdev=867.50, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=33.33% cpu : usr=0.02%, sys=21.25%, ctx=1705, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17704: Thu Apr 18 09:33:15 2024 read: IOPS=0, BW=835KiB/s (855kB/s)(2048KiB/2452msec) clat (msec): min=42, max=547, avg=294.98, stdev=357.11 lat (msec): min=42, max=547, avg=294.98, stdev=357.11 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max= 2048, per=48.68%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4176KiB/s (4276kB/s)(10.0MiB/2452msec) clat (msec): min=17, max=785, avg=185.87, stdev=281.13 lat (msec): min=17, max=785, avg=185.91, stdev=281.12 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 26], 40.00th=[ 30], 50.00th=[ 33], 60.00th=[ 90], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 634], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 2048, max= 4096, per=60.84%, avg=3584.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 3.50, stdev= 1.00, samples=4 lat (msec) : 20=16.67%, 50=33.33%, 100=8.33%, 250=16.67%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=12.32%, ctx=295, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17705: Thu Apr 18 09:33:15 2024 read: IOPS=1, BW=2030KiB/s (2078kB/s)(6144KiB/3027msec) clat (usec): min=345, max=789093, avg=234732.61, stdev=326911.99 lat (usec): min=346, max=789094, avg=234733.20, stdev=326911.96 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 40109], 50.00th=[ 40109], 60.00th=[ 98042], | 70.00th=[480248], 80.00th=[480248], 90.00th=[792724], 95.00th=[792724], | 99.00th=[792724], 99.50th=[792724], 99.90th=[792724], 99.95th=[792724], | 99.99th=[792724] bw ( KiB/s): min= 2043, max= 6144, per=81.10%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 write: IOPS=1, BW=2030KiB/s (2078kB/s)(6144KiB/3027msec) clat (msec): min=23, max=835, avg=269.40, stdev=377.99 lat (msec): min=23, max=835, avg=269.44, stdev=378.00 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 32], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2048, max= 4096, per=52.15%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 500=16.67% lat (msec) : 50=41.67%, 100=8.33%, 500=8.33%, 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=9.25%, ctx=586, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17706: Thu Apr 18 09:33:15 2024 read: IOPS=1, BW=1691KiB/s (1732kB/s)(5120KiB/3027msec) clat (usec): min=913, max=785052, avg=405152.01, stdev=364690.06 lat (usec): min=915, max=785053, avg=405153.14, stdev=364689.70 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 914], 10.00th=[ 914], 20.00th=[ 914], | 30.00th=[ 39584], 40.00th=[ 39584], 50.00th=[513803], 60.00th=[513803], | 70.00th=[683672], 80.00th=[683672], 90.00th=[784335], 95.00th=[784335], | 99.00th=[784335], 99.50th=[784335], 99.90th=[784335], 99.95th=[784335], | 99.99th=[784335] bw ( KiB/s): min= 2048, max= 4096, per=64.91%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=2, BW=2368KiB/s (2425kB/s)(7168KiB/3027msec) clat (msec): min=25, max=773, avg=142.77, stdev=278.53 lat (msec): min=25, max=773, avg=142.83, stdev=278.52 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 42], 60.00th=[ 45], | 70.00th=[ 45], 80.00th=[ 58], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 2048, max= 6144, per=81.12%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 1000=8.33% lat (msec) : 50=50.00%, 100=8.33%, 750=16.67%, 1000=16.67% cpu : usr=0.07%, sys=12.95%, ctx=901, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4208KiB/s (4309kB/s), 835KiB/s-2030KiB/s (855kB/s-2078kB/s), io=20.0MiB (20.0MB), run=2452-4867msec WRITE: bw=5891KiB/s (6032kB/s), 1052KiB/s-4176KiB/s (1077kB/s-4276kB/s), io=28.0MiB (29.4MB), run=2452-4867msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17708: Thu Apr 18 09:33:15 2024 read: IOPS=1, BW=1469KiB/s (1504kB/s)(7168KiB/4879msec) clat (msec): min=16, max=854, avg=440.96, stdev=401.99 lat (msec): min=16, max=854, avg=440.96, stdev=401.99 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 26], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 550], 60.00th=[ 760], | 70.00th=[ 760], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 1771, max= 2048, per=47.70%, avg=2001.83, stdev=113.08, samples=6 iops : min= 1, max= 2, avg= 1.83, stdev= 0.41, samples=6 write: IOPS=1, BW=1049KiB/s (1075kB/s)(5120KiB/4879msec) clat (msec): min=59, max=962, avg=358.08, stdev=376.76 lat (msec): min=59, max=962, avg=358.15, stdev=376.74 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 93], 40.00th=[ 93], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 1774, max= 4096, per=44.92%, avg=2639.33, stdev=1268.93, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 20=8.33%, 50=16.67%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33%, 1000=33.33% cpu : usr=0.04%, sys=4.26%, ctx=74, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17709: Thu Apr 18 09:33:15 2024 read: IOPS=0, BW=671KiB/s (687kB/s)(2048KiB/3053msec) clat (msec): min=19, max=506, avg=262.91, stdev=344.39 lat (msec): min=19, max=506, avg=262.91, stdev=344.39 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 20], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 2048, max= 2048, per=48.80%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3354KiB/s (3435kB/s)(10.0MiB/3053msec) clat (msec): min=20, max=813, avg=252.48, stdev=328.35 lat (msec): min=20, max=814, avg=252.53, stdev=328.34 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 31], | 30.00th=[ 38], 40.00th=[ 40], 50.00th=[ 45], 60.00th=[ 100], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 776], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 2039, max= 8192, per=69.67%, avg=4093.75, stdev=2898.43, samples=4 iops : min= 1, max= 8, avg= 3.75, stdev= 3.10, samples=4 lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=8.33%, 750=16.67% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=5.96%, ctx=131, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17710: Thu Apr 18 09:33:15 2024 read: IOPS=1, BW=1969KiB/s (2016kB/s)(6144KiB/3121msec) clat (msec): min=17, max=1088, avg=315.78, stdev=459.33 lat (msec): min=17, max=1088, avg=315.78, stdev=459.33 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 70], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2048, max= 4096, per=65.06%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1969KiB/s (2016kB/s)(6144KiB/3121msec) clat (msec): min=42, max=778, avg=203.13, stdev=288.79 lat (msec): min=42, max=778, avg=203.20, stdev=288.80 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 72], 50.00th=[ 72], 60.00th=[ 73], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 2048, max= 4096, per=43.57%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.03%, sys=4.71%, ctx=57, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17711: Thu Apr 18 09:33:15 2024 read: IOPS=1, BW=1613KiB/s (1652kB/s)(5120KiB/3174msec) clat (msec): min=17, max=699, avg=234.23, stdev=306.19 lat (msec): min=17, max=699, avg=234.24, stdev=306.19 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 4063, max= 4096, per=97.20%, avg=4079.50, stdev=23.33, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2258KiB/s (2313kB/s)(7168KiB/3174msec) clat (msec): min=25, max=829, avg=285.83, stdev=325.05 lat (msec): min=25, max=829, avg=285.88, stdev=325.05 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 52], | 30.00th=[ 75], 40.00th=[ 75], 50.00th=[ 140], 60.00th=[ 213], | 70.00th=[ 213], 80.00th=[ 667], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 2048, max= 6144, per=58.09%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=16.67%, 500=8.33% lat (msec) : 750=16.67%, 1000=8.33% cpu : usr=0.00%, sys=3.84%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4198KiB/s (4298kB/s), 671KiB/s-1969KiB/s (687kB/s-2016kB/s), io=20.0MiB (20.0MB), run=3053-4879msec WRITE: bw=5877KiB/s (6018kB/s), 1049KiB/s-3354KiB/s (1075kB/s-3435kB/s), io=28.0MiB (29.4MB), run=3053-4879msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17724: Thu Apr 18 09:33:18 2024 read: IOPS=1, BW=5922KiB/s (6064kB/s)(12.0MiB/2075msec) clat (msec): min=139, max=1527, avg=691.47, stdev=736.15 lat (msec): min=139, max=1527, avg=691.47, stdev=736.15 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 1536], 80.00th=[ 1536], 90.00th=[ 1536], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.05%, sys=44.26%, ctx=525, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17725: Thu Apr 18 09:33:18 2024 read: IOPS=0, BW=1818KiB/s (1862kB/s)(4096KiB/2253msec) clat (nsec): min=143963k, max=143963k, avg=143963485.00, stdev= 0.00 lat (nsec): min=143964k, max=143964k, avg=143964374.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3636KiB/s (3723kB/s)(8192KiB/2253msec) clat (msec): min=257, max=1850, avg=1053.95, stdev=1126.83 lat (msec): min=257, max=1850, avg=1054.11, stdev=1126.76 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=56.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=12.39%, ctx=315, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17726: Thu Apr 18 09:33:18 2024 write: IOPS=1, BW=5515KiB/s (5648kB/s)(12.0MiB/2228msec) clat (msec): min=203, max=1546, avg=742.16, stdev=709.95 lat (msec): min=204, max=1547, avg=742.41, stdev=709.96 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 477], 50.00th=[ 477], 60.00th=[ 477], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 8159, max= 8192, per=56.21%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.27%, ctx=133, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17727: Thu Apr 18 09:33:18 2024 write: IOPS=1, BW=6029KiB/s (6174kB/s)(12.0MiB/2038msec) clat (msec): min=142, max=1664, avg=676.64, stdev=856.20 lat (msec): min=143, max=1664, avg=676.86, stdev=856.23 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 1670], 80.00th=[ 1670], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 8094, max= 8175, per=55.93%, avg=8134.50, stdev=57.28, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=15.66%, ctx=68, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7272KiB/s (7447kB/s), 1818KiB/s-5922KiB/s (1862kB/s-6064kB/s), io=16.0MiB (16.8MB), run=2075-2253msec WRITE: bw=14.2MiB/s (14.9MB/s), 3636KiB/s-6029KiB/s (3723kB/s-6174kB/s), io=32.0MiB (33.6MB), run=2038-2253msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17720: Thu Apr 18 09:33:18 2024 read: IOPS=1, BW=5159KiB/s (5282kB/s)(12.0MiB/2382msec) clat (msec): min=217, max=1855, avg=793.63, stdev=920.80 lat (msec): min=217, max=1855, avg=793.63, stdev=920.80 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.81%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17721: Thu Apr 18 09:33:18 2024 read: IOPS=0, BW=1668KiB/s (1708kB/s)(4096KiB/2455msec) clat (nsec): min=55422k, max=55422k, avg=55421553.00, stdev= 0.00 lat (nsec): min=55424k, max=55424k, avg=55423861.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[55313], 5.00th=[55313], 10.00th=[55313], 20.00th=[55313], | 30.00th=[55313], 40.00th=[55313], 50.00th=[55313], 60.00th=[55313], | 70.00th=[55313], 80.00th=[55313], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3337KiB/s (3417kB/s)(8192KiB/2455msec) clat (msec): min=429, max=1963, avg=1196.54, stdev=1084.26 lat (msec): min=430, max=1963, avg=1196.78, stdev=1084.35 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 1972], | 70.00th=[ 1972], 80.00th=[ 1972], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 8175, max= 8175, per=63.24%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.44%, ctx=58, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17722: Thu Apr 18 09:33:18 2024 write: IOPS=1, BW=4847KiB/s (4964kB/s)(12.0MiB/2535msec) clat (msec): min=59, max=2043, avg=844.18, stdev=1055.38 lat (msec): min=59, max=2044, avg=844.44, stdev=1055.35 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 2039], 80.00th=[ 2039], 90.00th=[ 2039], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7684, max= 8192, per=61.41%, avg=7938.00, stdev=359.21, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.04%, sys=5.49%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17723: Thu Apr 18 09:33:18 2024 write: IOPS=5, BW=21.5MiB/s (22.6MB/s)(12.0MiB/558msec) clat (msec): min=62, max=373, avg=185.58, stdev=165.52 lat (msec): min=62, max=374, avg=185.79, stdev=165.57 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.36%, ctx=50, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6674KiB/s (6834kB/s), 1668KiB/s-5159KiB/s (1708kB/s-5282kB/s), io=16.0MiB (16.8MB), run=2382-2455msec WRITE: bw=12.6MiB/s (13.2MB/s), 3337KiB/s-21.5MiB/s (3417kB/s-22.6MB/s), io=32.0MiB (33.6MB), run=558-2535msec