== sanity test 398b: DIO and buffer IO race ============== 13:40:23 (1713289223) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.22537 s, 41.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14864: Tue Apr 16 13:40:37 2024 read: IOPS=128, BW=514KiB/s (526kB/s)(5992KiB/11667msec) clat (usec): min=56, max=160256, avg=4430.59, stdev=4657.38 lat (usec): min=57, max=160257, avg=4431.24, stdev=4657.37 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 1827], 10.00th=[ 2114], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3392], 50.00th=[ 3982], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[ 12518], 99.50th=[ 13829], 99.90th=[ 22152], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 296, max= 640, per=24.58%, avg=505.87, stdev=74.33, samples=23 iops : min= 74, max= 160, avg=126.39, stdev=18.58, samples=23 write: IOPS=134, BW=540KiB/s (553kB/s)(6296KiB/11667msec) clat (usec): min=426, max=18747, avg=3148.78, stdev=2793.20 lat (usec): min=426, max=18748, avg=3149.95, stdev=2792.81 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 775], 40.00th=[ 1893], 50.00th=[ 2671], 60.00th=[ 3163], | 70.00th=[ 3884], 80.00th=[ 5145], 90.00th=[ 7111], 95.00th=[ 8586], | 99.00th=[12256], 99.50th=[14353], 99.90th=[17957], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 360, max= 680, per=24.66%, avg=531.22, stdev=71.97, samples=23 iops : min= 90, max= 170, avg=132.74, stdev=17.96, samples=23 lat (usec) : 100=1.82%, 250=0.16%, 500=0.55%, 750=14.62%, 1000=2.18% lat (msec) : 2=5.24%, 4=36.43%, 10=36.33%, 20=2.60%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.20%, ctx=5930, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14865: Tue Apr 16 13:40:37 2024 read: IOPS=128, BW=514KiB/s (527kB/s)(5940KiB/11548msec) clat (usec): min=49, max=20830, avg=4312.80, stdev=2432.78 lat (usec): min=50, max=20830, avg=4313.62, stdev=2433.02 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1827], 10.00th=[ 2089], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4424], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[13960], 99.50th=[14877], 99.90th=[17695], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 328, max= 656, per=24.95%, avg=513.57, stdev=77.73, samples=23 iops : min= 82, max= 164, avg=128.30, stdev=19.49, samples=23 write: IOPS=137, BW=550KiB/s (563kB/s)(6348KiB/11548msec) clat (usec): min=457, max=158535, avg=3193.66, stdev=4680.13 lat (usec): min=457, max=158536, avg=3194.54, stdev=4680.14 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 766], 40.00th=[ 2114], 50.00th=[ 2638], 60.00th=[ 3163], | 70.00th=[ 4047], 80.00th=[ 5145], 90.00th=[ 6783], 95.00th=[ 8225], | 99.00th=[ 10814], 99.50th=[ 12256], 99.90th=[ 16188], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 368, max= 720, per=25.42%, avg=547.61, stdev=91.80, samples=23 iops : min= 92, max= 180, avg=136.83, stdev=22.98, samples=23 lat (usec) : 50=0.03%, 100=1.73%, 250=0.10%, 500=0.59%, 750=14.65% lat (usec) : 1000=2.41% lat (msec) : 2=4.49%, 4=37.11%, 10=36.78%, 20=2.05%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.25%, ctx=6017, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14866: Tue Apr 16 13:40:37 2024 read: IOPS=129, BW=519KiB/s (531kB/s)(6028KiB/11625msec) clat (usec): min=49, max=157229, avg=4327.71, stdev=4606.00 lat (usec): min=49, max=157230, avg=4329.30, stdev=4606.12 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 153], 10.00th=[ 2008], 20.00th=[ 2442], | 30.00th=[ 2835], 40.00th=[ 3359], 50.00th=[ 3916], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5866], 90.00th=[ 6915], 95.00th=[ 8225], | 99.00th=[ 13173], 99.50th=[ 14746], 99.90th=[ 19006], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 288, max= 696, per=25.11%, avg=516.70, stdev=81.34, samples=23 iops : min= 72, max= 174, avg=129.09, stdev=20.35, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6260KiB/11625msec) clat (usec): min=457, max=17999, avg=3212.47, stdev=2780.92 lat (usec): min=458, max=17999, avg=3213.52, stdev=2780.83 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 791], 40.00th=[ 2073], 50.00th=[ 2638], 60.00th=[ 3228], | 70.00th=[ 4113], 80.00th=[ 5342], 90.00th=[ 7242], 95.00th=[ 8586], | 99.00th=[11863], 99.50th=[13698], 99.90th=[15139], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 352, max= 736, per=24.81%, avg=534.35, stdev=94.36, samples=23 iops : min= 88, max= 184, avg=133.52, stdev=23.54, samples=23 lat (usec) : 50=0.07%, 100=2.15%, 250=0.33%, 500=0.42%, 750=14.32% lat (usec) : 1000=2.34% lat (msec) : 2=5.18%, 4=35.81%, 10=36.85%, 20=2.51%, 250=0.03% cpu : usr=0.21%, sys=23.31%, ctx=5945, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14868: Tue Apr 16 13:40:37 2024 read: IOPS=129, BW=519KiB/s (532kB/s)(6052KiB/11651msec) clat (usec): min=51, max=23368, avg=4349.80, stdev=2449.83 lat (usec): min=51, max=23368, avg=4350.35, stdev=2449.84 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 1713], 10.00th=[ 2147], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3359], 50.00th=[ 3949], 60.00th=[ 4555], | 70.00th=[ 5080], 80.00th=[ 6063], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[12780], 99.50th=[15533], 99.90th=[18482], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 232, max= 688, per=24.97%, avg=513.87, stdev=89.42, samples=23 iops : min= 58, max= 172, avg=128.39, stdev=22.38, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6236KiB/11651msec) clat (usec): min=417, max=162729, avg=3202.30, stdev=4845.48 lat (usec): min=418, max=162731, avg=3203.16, stdev=4845.53 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 603], | 30.00th=[ 742], 40.00th=[ 1745], 50.00th=[ 2606], 60.00th=[ 3228], | 70.00th=[ 4080], 80.00th=[ 5211], 90.00th=[ 6915], 95.00th=[ 8455], | 99.00th=[ 12125], 99.50th=[ 13042], 99.90th=[ 18220], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 296, max= 800, per=24.53%, avg=528.43, stdev=101.93, samples=23 iops : min= 74, max= 200, avg=132.04, stdev=25.42, samples=23 lat (usec) : 100=1.92%, 250=0.33%, 500=0.72%, 750=14.78%, 1000=2.21% lat (msec) : 2=4.72%, 4=35.48%, 10=37.57%, 20=2.21%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.92%, ctx=5976, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2058KiB/s (2108kB/s), 514KiB/s-519KiB/s (526kB/s-532kB/s), io=23.4MiB (24.6MB), run=11548-11667msec WRITE: bw=2155KiB/s (2207kB/s), 535KiB/s-550KiB/s (548kB/s-563kB/s), io=24.6MiB (25.7MB), run=11548-11667msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14869: Tue Apr 16 13:40:44 2024 read: IOPS=82, BW=328KiB/s (336kB/s)(5992KiB/18265msec) clat (usec): min=1619, max=162044, avg=4914.52, stdev=4980.32 lat (usec): min=1620, max=162046, avg=4915.42, stdev=4980.32 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2089], 10.00th=[ 2278], 20.00th=[ 2606], | 30.00th=[ 3032], 40.00th=[ 3621], 50.00th=[ 4047], 60.00th=[ 4359], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 9110], 95.00th=[ 11207], | 99.00th=[ 15533], 99.50th=[ 17171], 99.90th=[ 19268], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 200, max= 536, per=24.80%, avg=325.92, stdev=74.22, samples=36 iops : min= 50, max= 134, avg=81.42, stdev=18.57, samples=36 write: IOPS=86, BW=345KiB/s (353kB/s)(6296KiB/18265msec) clat (usec): min=2452, max=23434, avg=6892.04, stdev=3211.30 lat (usec): min=2453, max=23435, avg=6893.03, stdev=3211.36 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3228], 10.00th=[ 3523], 20.00th=[ 4080], | 30.00th=[ 4883], 40.00th=[ 5800], 50.00th=[ 6521], 60.00th=[ 6980], | 70.00th=[ 7570], 80.00th=[ 8848], 90.00th=[10945], 95.00th=[12780], | 99.00th=[18482], 99.50th=[21890], 99.90th=[22676], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 208, max= 592, per=24.85%, avg=341.92, stdev=71.76, samples=36 iops : min= 52, max= 148, avg=85.42, stdev=17.96, samples=36 lat (msec) : 2=1.60%, 4=31.74%, 10=56.58%, 20=9.67%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=12.29%, ctx=3612, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14870: Tue Apr 16 13:40:44 2024 read: IOPS=82, BW=330KiB/s (338kB/s)(5940KiB/18011msec) clat (usec): min=1707, max=162009, avg=4821.40, stdev=4955.10 lat (usec): min=1707, max=162010, avg=4822.22, stdev=4955.21 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2573], | 30.00th=[ 2900], 40.00th=[ 3523], 50.00th=[ 4015], 60.00th=[ 4293], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 8979], 95.00th=[ 10683], | 99.00th=[ 15401], 99.50th=[ 16909], 99.90th=[ 20055], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 200, max= 576, per=25.09%, avg=329.67, stdev=78.99, samples=36 iops : min= 50, max= 144, avg=82.36, stdev=19.68, samples=36 write: IOPS=88, BW=352KiB/s (361kB/s)(6348KiB/18011msec) clat (usec): min=2282, max=25635, avg=6802.58, stdev=3026.43 lat (usec): min=2283, max=25636, avg=6803.69, stdev=3026.45 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3195], 10.00th=[ 3523], 20.00th=[ 4178], | 30.00th=[ 4948], 40.00th=[ 5735], 50.00th=[ 6456], 60.00th=[ 6980], | 70.00th=[ 7635], 80.00th=[ 8586], 90.00th=[10552], 95.00th=[12387], | 99.00th=[16909], 99.50th=[20841], 99.90th=[25035], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 232, max= 576, per=25.61%, avg=352.33, stdev=78.53, samples=36 iops : min= 58, max= 144, avg=88.03, stdev=19.62, samples=36 lat (msec) : 2=1.40%, 4=31.22%, 10=57.78%, 20=9.28%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=12.39%, ctx=3579, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14871: Tue Apr 16 13:40:44 2024 read: IOPS=82, BW=331KiB/s (339kB/s)(6028KiB/18216msec) clat (usec): min=1598, max=20055, avg=4907.63, stdev=2938.90 lat (usec): min=1598, max=20055, avg=4908.43, stdev=2938.94 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2606], | 30.00th=[ 3064], 40.00th=[ 3720], 50.00th=[ 4113], 60.00th=[ 4490], | 70.00th=[ 5080], 80.00th=[ 6325], 90.00th=[ 9372], 95.00th=[10945], | 99.00th=[15401], 99.50th=[16909], 99.90th=[19530], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 176, max= 544, per=25.05%, avg=329.17, stdev=67.91, samples=36 iops : min= 44, max= 136, avg=82.17, stdev=16.95, samples=36 write: IOPS=85, BW=344KiB/s (352kB/s)(6260KiB/18216msec) clat (msec): min=2, max=167, avg= 6.88, stdev= 5.08 lat (msec): min=2, max=167, avg= 6.88, stdev= 5.08 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 24], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 216, max= 608, per=24.81%, avg=341.39, stdev=82.87, samples=36 iops : min= 54, max= 152, avg=85.22, stdev=20.73, samples=36 lat (msec) : 2=1.46%, 4=30.79%, 10=57.10%, 20=10.35%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=12.26%, ctx=3604, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14872: Tue Apr 16 13:40:44 2024 read: IOPS=83, BW=332KiB/s (340kB/s)(6052KiB/18225msec) clat (usec): min=1693, max=18545, avg=4808.56, stdev=2854.18 lat (usec): min=1693, max=18546, avg=4809.31, stdev=2854.19 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2089], 10.00th=[ 2245], 20.00th=[ 2573], | 30.00th=[ 2933], 40.00th=[ 3556], 50.00th=[ 4015], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 6390], 90.00th=[ 9241], 95.00th=[10814], | 99.00th=[14222], 99.50th=[15795], 99.90th=[17957], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 208, max= 584, per=25.11%, avg=330.00, stdev=81.30, samples=36 iops : min= 52, max= 146, avg=82.50, stdev=20.33, samples=36 write: IOPS=85, BW=342KiB/s (350kB/s)(6236KiB/18225msec) clat (msec): min=2, max=163, avg= 6.99, stdev= 5.11 lat (msec): min=2, max=163, avg= 7.00, stdev= 5.11 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 26], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 232, max= 592, per=24.73%, avg=340.22, stdev=74.91, samples=36 iops : min= 58, max= 148, avg=85.06, stdev=18.73, samples=36 lat (msec) : 2=1.69%, 4=31.38%, 10=56.05%, 20=10.38%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=12.24%, ctx=3565, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1315KiB/s (1346kB/s), 328KiB/s-332KiB/s (336kB/s-340kB/s), io=23.4MiB (24.6MB), run=18011-18265msec WRITE: bw=1376KiB/s (1409kB/s), 342KiB/s-352KiB/s (350kB/s-361kB/s), io=24.6MiB (25.7MB), run=18011-18265msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14899: Tue Apr 16 13:40:51 2024 read: IOPS=64, BW=1030KiB/s (1055kB/s)(5920KiB/5747msec) clat (usec): min=61, max=50812, avg=11235.33, stdev=3955.66 lat (usec): min=61, max=50813, avg=11235.92, stdev=3955.71 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 7242], 10.00th=[ 8160], 20.00th=[ 9110], | 30.00th=[ 9896], 40.00th=[10421], 50.00th=[11076], 60.00th=[11600], | 70.00th=[12256], 80.00th=[13435], 90.00th=[14877], 95.00th=[16188], | 99.00th=[22676], 99.50th=[29230], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 704, max= 1216, per=25.58%, avg=1026.45, stdev=136.00, samples=11 iops : min= 44, max= 76, avg=64.00, stdev= 8.50, samples=11 write: IOPS=69, BW=1108KiB/s (1135kB/s)(6368KiB/5747msec) clat (usec): min=741, max=18295, avg=3973.39, stdev=3543.43 lat (usec): min=742, max=18295, avg=3974.50, stdev=3543.43 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 816], 10.00th=[ 840], 20.00th=[ 1004], | 30.00th=[ 2147], 40.00th=[ 2474], 50.00th=[ 2737], 60.00th=[ 3458], | 70.00th=[ 4621], 80.00th=[ 5800], 90.00th=[ 8291], 95.00th=[12649], | 99.00th=[16581], 99.50th=[17695], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 800, max= 1472, per=26.59%, avg=1122.36, stdev=209.12, samples=11 iops : min= 50, max= 92, avg=70.00, stdev=12.95, samples=11 lat (usec) : 100=1.30%, 250=0.13%, 750=0.13%, 1000=9.90% lat (msec) : 2=4.82%, 4=19.40%, 10=27.99%, 20=35.68%, 50=0.52% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=19.11%, ctx=2652, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14901: Tue Apr 16 13:40:51 2024 read: IOPS=61, BW=990KiB/s (1014kB/s)(5744KiB/5800msec) clat (usec): min=72, max=28255, avg=10911.59, stdev=3006.04 lat (usec): min=72, max=28256, avg=10912.37, stdev=3006.11 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 7439], 10.00th=[ 8029], 20.00th=[ 8979], | 30.00th=[ 9503], 40.00th=[10159], 50.00th=[10683], 60.00th=[11338], | 70.00th=[11863], 80.00th=[12518], 90.00th=[14484], 95.00th=[16450], | 99.00th=[20841], 99.50th=[22414], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 608, max= 1184, per=24.72%, avg=991.82, stdev=150.18, samples=11 iops : min= 38, max= 74, avg=61.91, stdev= 9.43, samples=11 write: IOPS=70, BW=1128KiB/s (1155kB/s)(6544KiB/5800msec) clat (usec): min=740, max=82291, avg=4578.68, stdev=5529.63 lat (usec): min=741, max=82293, avg=4579.73, stdev=5529.63 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 824], 10.00th=[ 848], 20.00th=[ 1045], | 30.00th=[ 2180], 40.00th=[ 2573], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4883], 80.00th=[ 6587], 90.00th=[10290], 95.00th=[13173], | 99.00th=[16712], 99.50th=[17695], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 832, max= 1596, per=26.39%, avg=1113.82, stdev=266.55, samples=11 iops : min= 52, max= 99, avg=69.55, stdev=16.52, samples=11 lat (usec) : 100=0.52%, 250=0.13%, 750=0.39%, 1000=9.37% lat (msec) : 2=4.30%, 4=19.66%, 10=31.25%, 20=33.59%, 50=0.65% lat (msec) : 100=0.13% cpu : usr=0.03%, sys=18.71%, ctx=2602, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14903: Tue Apr 16 13:40:51 2024 read: IOPS=65, BW=1052KiB/s (1077kB/s)(6256KiB/5946msec) clat (usec): min=54, max=25659, avg=10764.75, stdev=3346.78 lat (usec): min=54, max=25660, avg=10765.52, stdev=3346.63 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 6915], 10.00th=[ 8029], 20.00th=[ 8717], | 30.00th=[ 9241], 40.00th=[ 9896], 50.00th=[10421], 60.00th=[11076], | 70.00th=[11731], 80.00th=[12780], 90.00th=[14484], 95.00th=[16057], | 99.00th=[20841], 99.50th=[24511], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 704, max= 1248, per=26.17%, avg=1049.91, stdev=152.73, samples=11 iops : min= 44, max= 78, avg=65.55, stdev= 9.56, samples=11 write: IOPS=63, BW=1014KiB/s (1039kB/s)(6032KiB/5946msec) clat (usec): min=707, max=68137, avg=4580.39, stdev=5299.57 lat (usec): min=708, max=68138, avg=4581.41, stdev=5299.57 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 807], 10.00th=[ 857], 20.00th=[ 1090], | 30.00th=[ 2212], 40.00th=[ 2573], 50.00th=[ 2966], 60.00th=[ 3589], | 70.00th=[ 5080], 80.00th=[ 6521], 90.00th=[ 9634], 95.00th=[14091], | 99.00th=[19792], 99.50th=[30802], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 544, max= 1248, per=23.43%, avg=988.82, stdev=218.30, samples=11 iops : min= 34, max= 78, avg=61.73, stdev=13.55, samples=11 lat (usec) : 100=1.17%, 250=0.13%, 750=0.78%, 1000=7.42% lat (msec) : 2=5.21%, 4=17.84%, 10=33.07%, 20=33.20%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=18.44%, ctx=2675, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14905: Tue Apr 16 13:40:51 2024 read: IOPS=63, BW=1011KiB/s (1035kB/s)(6032KiB/5969msec) clat (usec): min=54, max=21212, avg=10839.41, stdev=3375.65 lat (usec): min=54, max=21213, avg=10842.54, stdev=3375.80 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 6849], 10.00th=[ 7963], 20.00th=[ 8848], | 30.00th=[ 9503], 40.00th=[10159], 50.00th=[10683], 60.00th=[11338], | 70.00th=[12125], 80.00th=[13173], 90.00th=[14746], 95.00th=[15926], | 99.00th=[19268], 99.50th=[20579], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 672, max= 1120, per=24.86%, avg=997.55, stdev=125.22, samples=11 iops : min= 42, max= 70, avg=62.27, stdev= 7.77, samples=11 write: IOPS=65, BW=1048KiB/s (1073kB/s)(6256KiB/5969msec) clat (usec): min=673, max=69071, avg=4793.08, stdev=5544.96 lat (usec): min=674, max=69073, avg=4794.06, stdev=5545.01 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 799], 10.00th=[ 840], 20.00th=[ 1221], | 30.00th=[ 2180], 40.00th=[ 2606], 50.00th=[ 3097], 60.00th=[ 3982], | 70.00th=[ 5211], 80.00th=[ 6652], 90.00th=[10683], 95.00th=[14222], | 99.00th=[28705], 99.50th=[31327], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 768, max= 1312, per=24.94%, avg=1052.91, stdev=147.74, samples=11 iops : min= 48, max= 82, avg=65.73, stdev= 9.27, samples=11 lat (usec) : 100=1.56%, 250=0.26%, 750=0.78%, 1000=7.29% lat (msec) : 2=4.69%, 4=17.97%, 10=30.34%, 20=36.20%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=18.30%, ctx=2568, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4013KiB/s (4109kB/s), 990KiB/s-1052KiB/s (1014kB/s-1077kB/s), io=23.4MiB (24.5MB), run=5747-5969msec WRITE: bw=4222KiB/s (4323kB/s), 1014KiB/s-1128KiB/s (1039kB/s-1155kB/s), io=24.6MiB (25.8MB), run=5747-5969msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14900: Tue Apr 16 13:40:51 2024 read: IOPS=64, BW=1026KiB/s (1051kB/s)(5920KiB/5769msec) clat (usec): min=1892, max=51445, avg=6253.22, stdev=5749.62 lat (usec): min=1892, max=51446, avg=6253.80, stdev=5749.64 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2147], 10.00th=[ 2376], 20.00th=[ 2737], | 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 3982], 60.00th=[ 5342], | 70.00th=[ 6783], 80.00th=[ 9110], 90.00th=[12256], 95.00th=[15664], | 99.00th=[30540], 99.50th=[47973], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 704, max= 1280, per=25.62%, avg=1023.64, stdev=149.50, samples=11 iops : min= 44, max= 80, avg=63.91, stdev= 9.36, samples=11 write: IOPS=68, BW=1104KiB/s (1130kB/s)(6368KiB/5769msec) clat (usec): min=2687, max=26867, avg=8662.16, stdev=4806.99 lat (usec): min=2689, max=26868, avg=8663.49, stdev=4807.04 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3326], 10.00th=[ 3687], 20.00th=[ 4359], | 30.00th=[ 5473], 40.00th=[ 6325], 50.00th=[ 7439], 60.00th=[ 8586], | 70.00th=[10421], 80.00th=[12256], 90.00th=[16057], 95.00th=[18220], | 99.00th=[23987], 99.50th=[26084], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 736, max= 1376, per=26.57%, avg=1116.64, stdev=205.13, samples=11 iops : min= 46, max= 86, avg=69.73, stdev=12.82, samples=11 lat (msec) : 2=0.91%, 4=31.51%, 10=42.97%, 20=21.74%, 50=2.73% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=9.83%, ctx=1128, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14902: Tue Apr 16 13:40:51 2024 read: IOPS=70, BW=1126KiB/s (1154kB/s)(5744KiB/5099msec) clat (usec): min=1942, max=22290, avg=5288.17, stdev=3584.94 lat (usec): min=1943, max=22291, avg=5288.74, stdev=3584.94 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2769], 40.00th=[ 3195], 50.00th=[ 3720], 60.00th=[ 4817], | 70.00th=[ 6390], 80.00th=[ 8094], 90.00th=[10421], 95.00th=[12387], | 99.00th=[17433], 99.50th=[21627], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 736, max= 1472, per=28.20%, avg=1126.40, stdev=223.64, samples=10 iops : min= 46, max= 92, avg=70.40, stdev=13.98, samples=10 write: IOPS=80, BW=1283KiB/s (1314kB/s)(6544KiB/5099msec) clat (usec): min=2541, max=78467, avg=7809.03, stdev=5832.38 lat (usec): min=2543, max=78470, avg=7810.35, stdev=5832.42 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3228], 10.00th=[ 3392], 20.00th=[ 3785], | 30.00th=[ 4293], 40.00th=[ 5145], 50.00th=[ 6128], 60.00th=[ 6980], | 70.00th=[ 8848], 80.00th=[11338], 90.00th=[14877], 95.00th=[17695], | 99.00th=[23725], 99.50th=[25560], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1024, max= 1472, per=30.38%, avg=1276.80, stdev=131.90, samples=10 iops : min= 64, max= 92, avg=79.80, stdev= 8.24, samples=10 lat (msec) : 2=0.39%, 4=37.63%, 10=43.88%, 20=16.80%, 50=1.17% lat (msec) : 100=0.13% cpu : usr=0.04%, sys=9.93%, ctx=918, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14904: Tue Apr 16 13:40:51 2024 read: IOPS=65, BW=1044KiB/s (1069kB/s)(6256KiB/5995msec) clat (usec): min=1917, max=44331, avg=6267.05, stdev=4500.28 lat (usec): min=1917, max=44331, avg=6267.67, stdev=4500.27 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2180], 10.00th=[ 2409], 20.00th=[ 2737], | 30.00th=[ 3064], 40.00th=[ 3523], 50.00th=[ 4686], 60.00th=[ 6325], | 70.00th=[ 7570], 80.00th=[ 9503], 90.00th=[12518], 95.00th=[14484], | 99.00th=[17695], 99.50th=[23200], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 640, max= 1248, per=26.29%, avg=1050.18, stdev=163.68, samples=11 iops : min= 40, max= 78, avg=65.64, stdev=10.23, samples=11 write: IOPS=62, BW=1006KiB/s (1030kB/s)(6032KiB/5995msec) clat (usec): min=2800, max=59742, avg=9383.01, stdev=5737.84 lat (usec): min=2801, max=59744, avg=9384.22, stdev=5737.88 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3326], 10.00th=[ 3720], 20.00th=[ 4817], | 30.00th=[ 5866], 40.00th=[ 7177], 50.00th=[ 8225], 60.00th=[ 9241], | 70.00th=[10945], 80.00th=[12649], 90.00th=[16712], 95.00th=[19792], | 99.00th=[26346], 99.50th=[31589], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 640, max= 1248, per=23.53%, avg=989.09, stdev=186.29, samples=11 iops : min= 40, max= 78, avg=61.82, stdev=11.64, samples=11 lat (msec) : 2=0.78%, 4=27.99%, 10=44.92%, 20=23.70%, 50=2.47% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=9.93%, ctx=1195, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14906: Tue Apr 16 13:40:51 2024 read: IOPS=62, BW=1007KiB/s (1031kB/s)(6032KiB/5992msec) clat (usec): min=1745, max=50045, avg=6262.18, stdev=4684.15 lat (usec): min=1746, max=50045, avg=6262.82, stdev=4684.11 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2180], 10.00th=[ 2409], 20.00th=[ 2737], | 30.00th=[ 3261], 40.00th=[ 3687], 50.00th=[ 4555], 60.00th=[ 6259], | 70.00th=[ 7832], 80.00th=[ 9503], 90.00th=[11731], 95.00th=[13698], | 99.00th=[22152], 99.50th=[28967], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 704, max= 1213, per=24.82%, avg=991.73, stdev=140.47, samples=11 iops : min= 44, max= 75, avg=61.91, stdev= 8.65, samples=11 write: IOPS=65, BW=1044KiB/s (1069kB/s)(6256KiB/5992msec) clat (usec): min=2663, max=62326, avg=9263.13, stdev=5640.98 lat (usec): min=2664, max=62328, avg=9264.36, stdev=5640.99 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3359], 10.00th=[ 3818], 20.00th=[ 4686], | 30.00th=[ 5997], 40.00th=[ 7046], 50.00th=[ 8225], 60.00th=[ 9241], | 70.00th=[10814], 80.00th=[12518], 90.00th=[16450], 95.00th=[19268], | 99.00th=[26084], 99.50th=[30540], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 768, max= 1344, per=24.77%, avg=1041.27, stdev=157.72, samples=11 iops : min= 48, max= 84, avg=65.00, stdev= 9.89, samples=11 lat (msec) : 2=0.65%, 4=28.26%, 10=45.44%, 20=23.31%, 50=2.08% lat (msec) : 100=0.26% cpu : usr=0.07%, sys=10.08%, ctx=1200, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3995KiB/s (4091kB/s), 1007KiB/s-1126KiB/s (1031kB/s-1154kB/s), io=23.4MiB (24.5MB), run=5099-5995msec WRITE: bw=4204KiB/s (4304kB/s), 1006KiB/s-1283KiB/s (1030kB/s-1314kB/s), io=24.6MiB (25.8MB), run=5099-5995msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14916: Tue Apr 16 13:40:55 2024 read: IOPS=2, BW=2913KiB/s (2983kB/s)(7168KiB/2461msec) clat (msec): min=17, max=901, avg=150.45, stdev=331.18 lat (msec): min=17, max=901, avg=150.45, stdev=331.18 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 25], 60.00th=[ 33], | 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 4096, max= 6144, per=86.04%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2080KiB/s (2130kB/s)(5120KiB/2461msec) clat (msec): min=27, max=1230, avg=281.27, stdev=530.93 lat (msec): min=27, max=1230, avg=281.31, stdev=530.92 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 47], 60.00th=[ 47], | 70.00th=[ 55], 80.00th=[ 55], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 2048, max= 2048, per=24.58%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=66.67%, 100=8.33%, 1000=8.33% cpu : usr=0.00%, sys=4.76%, ctx=61, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14917: Tue Apr 16 13:40:55 2024 read: IOPS=0, BW=595KiB/s (609kB/s)(2048KiB/3441msec) clat (usec): min=32956, max=39036, avg=35996.61, stdev=4299.59 lat (usec): min=32957, max=39037, avg=35997.37, stdev=4299.69 clat percentiles (usec): | 1.00th=[32900], 5.00th=[32900], 10.00th=[32900], 20.00th=[32900], | 30.00th=[32900], 40.00th=[32900], 50.00th=[32900], 60.00th=[39060], | 70.00th=[39060], 80.00th=[39060], 90.00th=[39060], 95.00th=[39060], | 99.00th=[39060], 99.50th=[39060], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 2043, max= 2048, per=34.37%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2976KiB/s (3047kB/s)(10.0MiB/3441msec) clat (msec): min=27, max=1858, avg=336.71, stdev=588.55 lat (msec): min=27, max=1858, avg=336.77, stdev=588.54 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 41], 50.00th=[ 87], 60.00th=[ 111], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 844], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 4096, max=12263, per=98.17%, avg=8179.50, stdev=5774.94, samples=2 iops : min= 4, max= 11, avg= 7.50, stdev= 4.95, samples=2 lat (msec) : 50=50.00%, 100=8.33%, 250=25.00%, 1000=8.33% cpu : usr=0.00%, sys=4.07%, ctx=131, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14919: Tue Apr 16 13:40:55 2024 read: IOPS=1, BW=1862KiB/s (1907kB/s)(6144KiB/3300msec) clat (msec): min=44, max=645, avg=172.52, stdev=233.18 lat (msec): min=44, max=645, avg=172.53, stdev=233.18 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 75], 50.00th=[ 75], 60.00th=[ 97], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 8192, per=86.04%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=1, BW=1862KiB/s (1907kB/s)(6144KiB/3300msec) clat (msec): min=41, max=1893, avg=377.07, stdev=743.30 lat (msec): min=41, max=1893, avg=377.13, stdev=743.31 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 83], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 2048, max= 6144, per=49.16%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=25.00%, 100=41.67%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=5.61%, ctx=90, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14920: Tue Apr 16 13:40:55 2024 read: IOPS=1, BW=1560KiB/s (1598kB/s)(5120KiB/3281msec) clat (msec): min=24, max=803, avg=191.03, stdev=343.02 lat (msec): min=24, max=803, avg=191.03, stdev=343.02 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2185KiB/s (2237kB/s)(7168KiB/3281msec) clat (msec): min=40, max=1869, avg=331.76, stdev=679.01 lat (msec): min=41, max=1869, avg=331.83, stdev=679.03 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 64], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 142], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 6144, max= 8192, per=86.03%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 50=50.00%, 100=16.67%, 250=16.67%, 1000=8.33% cpu : usr=0.00%, sys=4.48%, ctx=59, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5952KiB/s (6095kB/s), 595KiB/s-2913KiB/s (609kB/s-2983kB/s), io=20.0MiB (20.0MB), run=2461-3441msec WRITE: bw=8332KiB/s (8532kB/s), 1862KiB/s-2976KiB/s (1907kB/s-3047kB/s), io=28.0MiB (29.4MB), run=2461-3441msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14921: Tue Apr 16 13:40:55 2024 read: IOPS=2, BW=2893KiB/s (2962kB/s)(7168KiB/2478msec) clat (usec): min=333, max=970590, avg=303562.66, stdev=438242.30 lat (usec): min=334, max=970592, avg=303563.56, stdev=438242.58 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 445], | 30.00th=[ 453], 40.00th=[ 453], 50.00th=[ 36439], 60.00th=[219153], | 70.00th=[219153], 80.00th=[901776], 90.00th=[968885], 95.00th=[968885], | 99.00th=[968885], 99.50th=[968885], 99.90th=[968885], 99.95th=[968885], | 99.99th=[968885] bw ( KiB/s): min= 2035, max= 2048, per=34.32%, avg=2043.67, stdev= 7.51, samples=3 iops : min= 1, max= 2, avg= 1.67, stdev= 0.58, samples=3 write: IOPS=2, BW=2066KiB/s (2116kB/s)(5120KiB/2478msec) clat (msec): min=28, max=135, avg=69.60, stdev=47.99 lat (msec): min=28, max=135, avg=69.65, stdev=47.98 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 2035, max= 2035, per=24.41%, avg=2035.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 50=25.00%, 100=8.33%, 250=25.00%, 1000=16.67% cpu : usr=0.00%, sys=19.66%, ctx=675, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14922: Tue Apr 16 13:40:55 2024 read: IOPS=0, BW=596KiB/s (610kB/s)(2048KiB/3439msec) clat (msec): min=105, max=800, avg=452.85, stdev=491.03 lat (msec): min=105, max=800, avg=452.85, stdev=491.03 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 802], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 1787, max= 2048, per=32.20%, avg=1917.50, stdev=184.55, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2978KiB/s (3049kB/s)(10.0MiB/3439msec) clat (msec): min=26, max=1878, avg=253.21, stdev=575.30 lat (msec): min=26, max=1878, avg=253.25, stdev=575.30 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 29], 40.00th=[ 31], 50.00th=[ 36], 60.00th=[ 41], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 239], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 2019, max= 7160, per=58.13%, avg=4846.67, stdev=2608.81, samples=3 iops : min= 1, max= 6, avg= 4.00, stdev= 2.65, samples=3 lat (msec) : 50=50.00%, 100=8.33%, 250=25.00%, 1000=8.33% cpu : usr=0.03%, sys=27.34%, ctx=230, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14923: Tue Apr 16 13:40:55 2024 read: IOPS=2, BW=2464KiB/s (2523kB/s)(6144KiB/2494msec) clat (usec): min=346, max=76193, avg=30903.96, stdev=35260.17 lat (usec): min=347, max=76194, avg=30904.65, stdev=35260.17 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 371], 50.00th=[ 371], 60.00th=[42206], | 70.00th=[66323], 80.00th=[66323], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] write: IOPS=2, BW=2464KiB/s (2523kB/s)(6144KiB/2494msec) clat (msec): min=29, max=1941, avg=384.61, stdev=763.31 lat (msec): min=29, max=1941, avg=384.63, stdev=763.32 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 75], 50.00th=[ 75], 60.00th=[ 91], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 4096, max= 4096, per=49.13%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 50=25.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=9.79%, ctx=140, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14924: Tue Apr 16 13:40:55 2024 read: IOPS=2, BW=2066KiB/s (2116kB/s)(5120KiB/2478msec) clat (usec): min=298, max=30720, avg=6727.87, stdev=13431.09 lat (usec): min=299, max=30720, avg=6728.41, stdev=13431.19 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 297], 10.00th=[ 297], 20.00th=[ 297], | 30.00th=[ 310], 40.00th=[ 310], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 1958], 80.00th=[ 1958], 90.00th=[30802], 95.00th=[30802], | 99.00th=[30802], 99.50th=[30802], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] write: IOPS=2, BW=2893KiB/s (2962kB/s)(7168KiB/2478msec) clat (msec): min=32, max=1929, avg=348.74, stdev=698.26 lat (msec): min=32, max=1930, avg=348.79, stdev=698.26 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 52], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 86], 60.00th=[ 113], | 70.00th=[ 113], 80.00th=[ 146], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 2048, max= 2048, per=24.57%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 2=8.33%, 50=16.67%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=8.07%, ctx=56, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5955KiB/s (6098kB/s), 596KiB/s-2893KiB/s (610kB/s-2962kB/s), io=20.0MiB (20.0MB), run=2478-3439msec WRITE: bw=8337KiB/s (8537kB/s), 2066KiB/s-2978KiB/s (2116kB/s-3049kB/s), io=28.0MiB (29.4MB), run=2478-3439msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14932: Tue Apr 16 13:40:58 2024 read: IOPS=2, BW=9239KiB/s (9461kB/s)(12.0MiB/1330msec) clat (usec): min=1997, max=1161.6k, avg=442550.20, stdev=628001.24 lat (usec): min=1999, max=1161.6k, avg=442551.29, stdev=628000.58 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 1991], 10.00th=[ 1991], | 20.00th=[ 1991], 30.00th=[ 1991], 40.00th=[ 164627], | 50.00th=[ 164627], 60.00th=[ 164627], 70.00th=[1166017], | 80.00th=[1166017], 90.00th=[1166017], 95.00th=[1166017], | 99.00th=[1166017], 99.50th=[1166017], 99.90th=[1166017], | 99.95th=[1166017], 99.99th=[1166017] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 2=33.33%, 250=33.33% cpu : usr=0.00%, sys=29.27%, ctx=589, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14933: Tue Apr 16 13:40:58 2024 read: IOPS=0, BW=2384KiB/s (2441kB/s)(4096KiB/1718msec) clat (nsec): min=211848k, max=211848k, avg=211847906.00, stdev= 0.00 lat (nsec): min=211850k, max=211850k, avg=211849581.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 8143, max= 8143, per=85.39%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4768KiB/s (4883kB/s)(8192KiB/1718msec) clat (msec): min=333, max=1163, avg=748.62, stdev=587.40 lat (msec): min=333, max=1164, avg=749.02, stdev=587.28 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 1167], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7831, max= 7831, per=41.06%, avg=7831.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=22.07%, ctx=371, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14934: Tue Apr 16 13:40:58 2024 write: IOPS=1, BW=7543KiB/s (7724kB/s)(12.0MiB/1629msec) clat (msec): min=133, max=1226, avg=539.85, stdev=598.01 lat (msec): min=133, max=1226, avg=540.09, stdev=598.09 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min=16286, max=16286, per=85.39%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=20.64%, ctx=68, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14935: Tue Apr 16 13:40:58 2024 write: IOPS=5, BW=21.1MiB/s (22.1MB/s)(12.0MiB/570msec) clat (msec): min=108, max=346, avg=188.19, stdev=137.53 lat (msec): min=108, max=347, avg=188.43, stdev=137.65 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min=16189, max=16189, per=84.88%, avg=16189.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=43.06%, ctx=52, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9537KiB/s (9766kB/s), 2384KiB/s-9239KiB/s (2441kB/s-9461kB/s), io=16.0MiB (16.8MB), run=1330-1718msec WRITE: bw=18.6MiB/s (19.5MB/s), 4768KiB/s-21.1MiB/s (4883kB/s-22.1MB/s), io=32.0MiB (33.6MB), run=570-1718msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14936: Tue Apr 16 13:40:58 2024 read: IOPS=1, BW=6715KiB/s (6876kB/s)(12.0MiB/1830msec) clat (msec): min=294, max=1146, avg=609.40, stdev=467.64 lat (msec): min=294, max=1146, avg=609.40, stdev=467.64 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 5490, max= 8192, per=83.85%, avg=6841.00, stdev=1910.60, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=5.36%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14937: Tue Apr 16 13:40:58 2024 read: IOPS=0, BW=2040KiB/s (2089kB/s)(4096KiB/2008msec) clat (nsec): min=391731k, max=391731k, avg=391731435.00, stdev= 0.00 lat (nsec): min=391733k, max=391733k, avg=391732738.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=4080KiB/s (4178kB/s)(8192KiB/2008msec) clat (msec): min=159, max=1437, avg=798.47, stdev=903.05 lat (msec): min=160, max=1437, avg=798.74, stdev=902.98 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 1435], | 70.00th=[ 1435], 80.00th=[ 1435], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.05%, sys=7.82%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14938: Tue Apr 16 13:40:58 2024 write: IOPS=1, BW=6830KiB/s (6994kB/s)(12.0MiB/1799msec) clat (msec): min=327, max=1109, avg=595.14, stdev=445.67 lat (msec): min=328, max=1109, avg=595.45, stdev=445.51 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8015, max= 8192, per=49.66%, avg=8103.50, stdev=125.16, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.17%, sys=9.40%, ctx=37, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14939: Tue Apr 16 13:40:58 2024 write: IOPS=4, BW=16.9MiB/s (17.7MB/s)(12.0MiB/710msec) clat (msec): min=211, max=264, avg=234.99, stdev=27.09 lat (msec): min=212, max=265, avg=235.21, stdev=27.13 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=32.16%, ctx=73, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8159KiB/s (8355kB/s), 2040KiB/s-6715KiB/s (2089kB/s-6876kB/s), io=16.0MiB (16.8MB), run=1830-2008msec WRITE: bw=15.9MiB/s (16.7MB/s), 4080KiB/s-16.9MiB/s (4178kB/s-17.7MB/s), io=32.0MiB (33.6MB), run=710-2008msec