== sanity test 398b: DIO and buffer IO race ============== 12:53:23 (1713372803) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.24293 s, 40.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18956: Wed Apr 17 12:53:38 2024 read: IOPS=122, BW=488KiB/s (500kB/s)(5992KiB/12275msec) clat (usec): min=52, max=166967, avg=4576.31, stdev=4905.92 lat (usec): min=52, max=166969, avg=4577.48, stdev=4905.73 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 118], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3490], 50.00th=[ 4080], 60.00th=[ 4752], | 70.00th=[ 5473], 80.00th=[ 6194], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[ 12780], 99.50th=[ 15401], 99.90th=[ 28705], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 264, max= 584, per=25.03%, avg=481.58, stdev=68.79, samples=24 iops : min= 66, max= 146, avg=120.25, stdev=17.13, samples=24 write: IOPS=128, BW=513KiB/s (525kB/s)(6296KiB/12275msec) clat (usec): min=443, max=24778, avg=3394.14, stdev=3092.91 lat (usec): min=444, max=24779, avg=3395.13, stdev=3093.24 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 660], | 30.00th=[ 865], 40.00th=[ 1975], 50.00th=[ 2737], 60.00th=[ 3425], | 70.00th=[ 4359], 80.00th=[ 5538], 90.00th=[ 7504], 95.00th=[ 9372], | 99.00th=[13435], 99.50th=[15926], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 312, max= 704, per=25.19%, avg=507.25, stdev=83.38, samples=24 iops : min= 78, max= 176, avg=126.67, stdev=20.82, samples=24 lat (usec) : 100=2.08%, 250=0.65%, 500=0.42%, 750=12.43%, 1000=3.97% lat (msec) : 2=4.79%, 4=33.72%, 10=38.64%, 20=3.09%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.50%, ctx=5734, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18958: Wed Apr 17 12:53:38 2024 read: IOPS=119, BW=478KiB/s (490kB/s)(5940KiB/12419msec) clat (usec): min=50, max=25713, avg=4482.89, stdev=2536.79 lat (usec): min=50, max=25714, avg=4484.33, stdev=2536.93 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 367], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 3982], 60.00th=[ 4621], | 70.00th=[ 5276], 80.00th=[ 6194], 90.00th=[ 7635], 95.00th=[ 8848], | 99.00th=[13173], 99.50th=[15533], 99.90th=[19268], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 256, max= 608, per=24.69%, avg=475.12, stdev=77.62, samples=24 iops : min= 64, max= 152, avg=118.71, stdev=19.36, samples=24 write: IOPS=127, BW=511KiB/s (523kB/s)(6348KiB/12419msec) clat (usec): min=410, max=165379, avg=3580.87, stdev=5113.24 lat (usec): min=410, max=165380, avg=3583.32, stdev=5112.69 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 685], | 30.00th=[ 938], 40.00th=[ 2073], 50.00th=[ 2802], 60.00th=[ 3490], | 70.00th=[ 4490], 80.00th=[ 5932], 90.00th=[ 7635], 95.00th=[ 9241], | 99.00th=[ 13304], 99.50th=[ 15533], 99.90th=[ 28705], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 304, max= 608, per=24.85%, avg=500.42, stdev=77.16, samples=24 iops : min= 76, max= 152, avg=125.04, stdev=19.23, samples=24 lat (usec) : 100=1.89%, 250=0.46%, 500=0.72%, 750=11.23%, 1000=4.56% lat (msec) : 2=4.88%, 4=34.51%, 10=38.44%, 20=3.19%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.16%, ctx=5908, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18960: Wed Apr 17 12:53:38 2024 read: IOPS=120, BW=483KiB/s (495kB/s)(6028KiB/12479msec) clat (usec): min=49, max=26517, avg=4596.97, stdev=2540.09 lat (usec): min=50, max=26518, avg=4597.78, stdev=2540.28 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 141], 10.00th=[ 2212], 20.00th=[ 2671], | 30.00th=[ 3064], 40.00th=[ 3621], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5538], 80.00th=[ 6390], 90.00th=[ 7701], 95.00th=[ 8979], | 99.00th=[12125], 99.50th=[14222], 99.90th=[20317], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 240, max= 592, per=24.91%, avg=479.29, stdev=71.61, samples=24 iops : min= 60, max= 148, avg=119.67, stdev=17.89, samples=24 write: IOPS=125, BW=502KiB/s (514kB/s)(6260KiB/12479msec) clat (usec): min=413, max=174578, avg=3495.28, stdev=5251.05 lat (usec): min=414, max=174579, avg=3496.89, stdev=5251.04 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 652], | 30.00th=[ 840], 40.00th=[ 1958], 50.00th=[ 2769], 60.00th=[ 3458], | 70.00th=[ 4293], 80.00th=[ 5735], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[ 12518], 99.50th=[ 14484], 99.90th=[ 20317], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 312, max= 648, per=24.64%, avg=496.21, stdev=70.01, samples=24 iops : min= 78, max= 162, avg=123.92, stdev=17.47, samples=24 lat (usec) : 50=0.03%, 100=2.02%, 250=0.52%, 500=0.46%, 750=12.96% lat (usec) : 1000=3.87% lat (msec) : 2=4.13%, 4=33.24%, 10=39.71%, 20=2.93%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.94%, ctx=5917, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18961: Wed Apr 17 12:53:38 2024 read: IOPS=122, BW=488KiB/s (500kB/s)(6052KiB/12393msec) clat (usec): min=50, max=17297, avg=4438.97, stdev=2432.49 lat (usec): min=50, max=17297, avg=4439.58, stdev=2432.51 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 123], 10.00th=[ 2114], 20.00th=[ 2638], | 30.00th=[ 3064], 40.00th=[ 3523], 50.00th=[ 4047], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6194], 90.00th=[ 7373], 95.00th=[ 8586], | 99.00th=[12518], 99.50th=[13960], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 240, max= 600, per=25.06%, avg=482.25, stdev=73.39, samples=24 iops : min= 60, max= 150, avg=120.50, stdev=18.36, samples=24 write: IOPS=125, BW=503KiB/s (515kB/s)(6236KiB/12393msec) clat (usec): min=396, max=174719, avg=3587.87, stdev=5420.60 lat (usec): min=396, max=174721, avg=3590.50, stdev=5420.68 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 553], 20.00th=[ 627], | 30.00th=[ 848], 40.00th=[ 1565], 50.00th=[ 2769], 60.00th=[ 3458], | 70.00th=[ 4490], 80.00th=[ 5997], 90.00th=[ 7963], 95.00th=[ 9765], | 99.00th=[ 13435], 99.50th=[ 16319], 99.90th=[ 32113], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 296, max= 752, per=24.75%, avg=498.54, stdev=80.57, samples=24 iops : min= 74, max= 188, avg=124.58, stdev=20.15, samples=24 lat (usec) : 100=2.08%, 250=0.72%, 500=0.49%, 750=13.51%, 1000=3.12% lat (msec) : 2=5.44%, 4=32.45%, 10=38.54%, 20=3.52%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.09%, ctx=5796, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1924KiB/s (1970kB/s), 478KiB/s-488KiB/s (490kB/s-500kB/s), io=23.4MiB (24.6MB), run=12275-12479msec WRITE: bw=2015KiB/s (2063kB/s), 502KiB/s-513KiB/s (514kB/s-525kB/s), io=24.6MiB (25.7MB), run=12275-12479msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18954: Wed Apr 17 12:53:42 2024 read: IOPS=94, BW=378KiB/s (387kB/s)(5992KiB/15852msec) clat (usec): min=1487, max=197291, avg=4318.43, stdev=5912.48 lat (usec): min=1488, max=197292, avg=4319.06, stdev=5912.49 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2073], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3294], | 70.00th=[ 4228], 80.00th=[ 5866], 90.00th=[ 8979], 95.00th=[ 11207], | 99.00th=[ 16188], 99.50th=[ 18482], 99.90th=[ 21627], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 144, max= 784, per=24.87%, avg=371.06, stdev=173.08, samples=31 iops : min= 36, max= 196, avg=92.74, stdev=43.28, samples=31 write: IOPS=99, BW=397KiB/s (407kB/s)(6296KiB/15852msec) clat (usec): min=2209, max=29439, avg=5914.15, stdev=3511.18 lat (usec): min=2210, max=29440, avg=5914.95, stdev=3511.24 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2802], 10.00th=[ 2966], 20.00th=[ 3228], | 30.00th=[ 3490], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5473], | 70.00th=[ 7046], 80.00th=[ 8586], 90.00th=[10814], 95.00th=[12649], | 99.00th=[18482], 99.50th=[20841], 99.90th=[26870], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 160, max= 848, per=24.85%, avg=388.10, stdev=176.51, samples=31 iops : min= 40, max= 212, avg=97.00, stdev=44.14, samples=31 lat (msec) : 2=7.16%, 4=47.69%, 10=35.03%, 20=9.64%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.19%, ctx=3579, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18955: Wed Apr 17 12:53:42 2024 read: IOPS=92, BW=369KiB/s (378kB/s)(5940KiB/16093msec) clat (usec): min=1495, max=169713, avg=4331.39, stdev=5418.58 lat (usec): min=1496, max=169716, avg=4332.05, stdev=5418.60 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2057], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 4146], 80.00th=[ 5866], 90.00th=[ 9241], 95.00th=[ 11469], | 99.00th=[ 16319], 99.50th=[ 19530], 99.90th=[ 23987], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 96, max= 776, per=24.06%, avg=358.97, stdev=183.32, samples=31 iops : min= 24, max= 194, avg=89.74, stdev=45.83, samples=31 write: IOPS=98, BW=394KiB/s (404kB/s)(6348KiB/16093msec) clat (usec): min=2191, max=29444, avg=6041.91, stdev=3816.93 lat (usec): min=2191, max=29446, avg=6042.79, stdev=3816.86 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2769], 10.00th=[ 2999], 20.00th=[ 3261], | 30.00th=[ 3523], 40.00th=[ 3884], 50.00th=[ 4359], 60.00th=[ 5342], | 70.00th=[ 6980], 80.00th=[ 8848], 90.00th=[11076], 95.00th=[12911], | 99.00th=[20055], 99.50th=[23462], 99.90th=[28705], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 160, max= 800, per=24.45%, avg=381.94, stdev=172.81, samples=31 iops : min= 40, max= 200, avg=95.48, stdev=43.20, samples=31 lat (msec) : 2=7.42%, 4=47.92%, 10=33.66%, 20=10.32%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.18%, ctx=3577, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18957: Wed Apr 17 12:53:42 2024 read: IOPS=94, BW=376KiB/s (385kB/s)(6028KiB/16016msec) clat (usec): min=1521, max=23419, avg=4244.97, stdev=3205.08 lat (usec): min=1522, max=23419, avg=4245.57, stdev=3205.07 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1926], 20.00th=[ 2089], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 5997], 90.00th=[ 9110], 95.00th=[11338], | 99.00th=[15270], 99.50th=[18220], 99.90th=[20317], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 120, max= 808, per=24.54%, avg=366.19, stdev=171.83, samples=31 iops : min= 30, max= 202, avg=91.55, stdev=42.96, samples=31 write: IOPS=97, BW=391KiB/s (400kB/s)(6260KiB/16016msec) clat (msec): min=2, max=173, avg= 6.10, stdev= 5.64 lat (msec): min=2, max=173, avg= 6.10, stdev= 5.64 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 23], 99.90th=[ 28], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 168, max= 824, per=24.24%, avg=378.58, stdev=179.97, samples=31 iops : min= 42, max= 206, avg=94.65, stdev=44.99, samples=31 lat (msec) : 2=6.97%, 4=48.73%, 10=33.72%, 20=9.93%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.22%, ctx=3616, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18959: Wed Apr 17 12:53:42 2024 read: IOPS=94, BW=379KiB/s (388kB/s)(6052KiB/15960msec) clat (usec): min=1551, max=22243, avg=4251.91, stdev=3347.70 lat (usec): min=1552, max=22244, avg=4252.53, stdev=3347.71 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1909], 20.00th=[ 2040], | 30.00th=[ 2180], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 9503], 95.00th=[11731], | 99.00th=[15664], 99.50th=[19268], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 128, max= 792, per=24.71%, avg=368.74, stdev=173.97, samples=31 iops : min= 32, max= 198, avg=92.16, stdev=43.50, samples=31 write: IOPS=97, BW=391KiB/s (400kB/s)(6236KiB/15960msec) clat (msec): min=2, max=174, avg= 6.06, stdev= 5.62 lat (msec): min=2, max=174, avg= 6.06, stdev= 5.62 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 28], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 160, max= 768, per=24.30%, avg=379.58, stdev=175.52, samples=31 iops : min= 40, max= 192, avg=94.87, stdev=43.89, samples=31 lat (msec) : 2=7.98%, 4=47.92%, 10=32.71%, 20=10.77%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.32%, ctx=3634, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1492KiB/s (1528kB/s), 369KiB/s-379KiB/s (378kB/s-388kB/s), io=23.4MiB (24.6MB), run=15852-16093msec WRITE: bw=1562KiB/s (1600kB/s), 391KiB/s-397KiB/s (400kB/s-407kB/s), io=24.6MiB (25.7MB), run=15852-16093msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18991: Wed Apr 17 12:53:49 2024 read: IOPS=64, BW=1027KiB/s (1052kB/s)(5920KiB/5762msec) clat (usec): min=64, max=78427, avg=11395.31, stdev=5116.86 lat (usec): min=65, max=78428, avg=11395.88, stdev=5116.89 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 2376], 10.00th=[ 8029], 20.00th=[ 8848], | 30.00th=[ 9896], 40.00th=[10683], 50.00th=[11338], 60.00th=[12125], | 70.00th=[12911], 80.00th=[13960], 90.00th=[15008], 95.00th=[16712], | 99.00th=[20317], 99.50th=[25297], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 736, max= 1152, per=24.47%, avg=991.82, stdev=107.21, samples=11 iops : min= 46, max= 72, avg=61.91, stdev= 6.76, samples=11 write: IOPS=69, BW=1105KiB/s (1132kB/s)(6368KiB/5762msec) clat (usec): min=675, max=23498, avg=3865.23, stdev=3403.90 lat (usec): min=676, max=23499, avg=3866.16, stdev=3403.89 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 930], | 30.00th=[ 2212], 40.00th=[ 2606], 50.00th=[ 2966], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5473], 90.00th=[ 8029], 95.00th=[11731], | 99.00th=[16581], 99.50th=[17695], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 672, max= 1408, per=25.30%, avg=1079.00, stdev=255.65, samples=11 iops : min= 42, max= 88, avg=67.36, stdev=15.88, samples=11 lat (usec) : 100=2.08%, 500=0.13%, 750=0.78%, 1000=10.94% lat (msec) : 2=3.26%, 4=20.05%, 10=25.78%, 20=36.33%, 50=0.52% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=20.26%, ctx=2534, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18992: Wed Apr 17 12:53:49 2024 read: IOPS=62, BW=1004KiB/s (1028kB/s)(5744KiB/5720msec) clat (usec): min=60, max=26422, avg=11460.67, stdev=4169.77 lat (usec): min=61, max=26422, avg=11462.65, stdev=4169.71 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 75], 10.00th=[ 8225], 20.00th=[ 9241], | 30.00th=[10028], 40.00th=[10814], 50.00th=[11469], 60.00th=[12125], | 70.00th=[13173], 80.00th=[14222], 90.00th=[15664], 95.00th=[17171], | 99.00th=[22676], 99.50th=[24511], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 672, max= 1152, per=24.10%, avg=977.00, stdev=130.46, samples=11 iops : min= 42, max= 72, avg=60.91, stdev= 8.10, samples=11 write: IOPS=71, BW=1144KiB/s (1172kB/s)(6544KiB/5720msec) clat (usec): min=667, max=92900, avg=3904.45, stdev=5610.36 lat (usec): min=667, max=92902, avg=3905.32, stdev=5610.40 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 807], 20.00th=[ 947], | 30.00th=[ 1467], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3097], | 70.00th=[ 3884], 80.00th=[ 5276], 90.00th=[ 8291], 95.00th=[12125], | 99.00th=[16712], 99.50th=[18482], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 832, max= 1408, per=25.98%, avg=1107.91, stdev=200.62, samples=11 iops : min= 52, max= 88, avg=69.09, stdev=12.61, samples=11 lat (usec) : 100=2.60%, 250=0.13%, 500=0.13%, 750=1.17%, 1000=10.68% lat (msec) : 2=5.73%, 4=20.57%, 10=22.01%, 20=35.55%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=19.48%, ctx=2460, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18993: Wed Apr 17 12:53:49 2024 read: IOPS=66, BW=1059KiB/s (1084kB/s)(6256KiB/5908msec) clat (usec): min=63, max=20490, avg=10840.21, stdev=3877.59 lat (usec): min=63, max=20490, avg=10841.52, stdev=3878.16 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 77], 10.00th=[ 7570], 20.00th=[ 9241], | 30.00th=[10028], 40.00th=[10814], 50.00th=[11207], 60.00th=[11863], | 70.00th=[12649], 80.00th=[13304], 90.00th=[14615], 95.00th=[15926], | 99.00th=[19006], 99.50th=[20317], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 736, max= 1216, per=25.61%, avg=1038.27, stdev=131.41, samples=11 iops : min= 46, max= 76, avg=64.82, stdev= 8.21, samples=11 write: IOPS=63, BW=1021KiB/s (1045kB/s)(6032KiB/5908msec) clat (usec): min=671, max=98016, avg=4405.62, stdev=6261.35 lat (usec): min=671, max=98018, avg=4409.47, stdev=6261.62 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 824], 20.00th=[ 979], | 30.00th=[ 1696], 40.00th=[ 2573], 50.00th=[ 3032], 60.00th=[ 3458], | 70.00th=[ 4359], 80.00th=[ 6194], 90.00th=[ 8848], 95.00th=[14222], | 99.00th=[20055], 99.50th=[22152], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 704, max= 1248, per=23.18%, avg=988.82, stdev=176.95, samples=11 iops : min= 44, max= 78, avg=61.73, stdev=10.97, samples=11 lat (usec) : 100=2.99%, 250=0.78%, 750=1.30%, 1000=9.11% lat (msec) : 2=4.69%, 4=17.84%, 10=22.53%, 20=39.84%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.00%, sys=19.25%, ctx=2512, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18994: Wed Apr 17 12:53:49 2024 read: IOPS=66, BW=1057KiB/s (1083kB/s)(6032KiB/5705msec) clat (usec): min=51, max=22904, avg=11221.32, stdev=3765.12 lat (usec): min=51, max=22905, avg=11221.95, stdev=3765.15 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 91], 10.00th=[ 8029], 20.00th=[ 9241], | 30.00th=[10159], 40.00th=[10945], 50.00th=[11731], 60.00th=[12256], | 70.00th=[12911], 80.00th=[13698], 90.00th=[14877], 95.00th=[16057], | 99.00th=[19530], 99.50th=[21365], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 766, max= 1184, per=25.53%, avg=1035.18, stdev=126.10, samples=11 iops : min= 47, max= 74, avg=64.55, stdev= 8.04, samples=11 write: IOPS=68, BW=1097KiB/s (1123kB/s)(6256KiB/5705msec) clat (usec): min=666, max=108132, avg=3750.45, stdev=6102.78 lat (usec): min=666, max=108134, avg=3751.32, stdev=6102.82 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 947], | 30.00th=[ 1696], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3228], | 70.00th=[ 3654], 80.00th=[ 4817], 90.00th=[ 7373], 95.00th=[ 9503], | 99.00th=[ 16319], 99.50th=[ 22938], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 736, max= 1440, per=25.43%, avg=1084.64, stdev=215.97, samples=11 iops : min= 46, max= 90, avg=67.64, stdev=13.58, samples=11 lat (usec) : 100=2.47%, 250=0.52%, 750=0.91%, 1000=10.03% lat (msec) : 2=4.95%, 4=21.87%, 10=21.61%, 20=37.11%, 50=0.39% lat (msec) : 250=0.13% cpu : usr=0.07%, sys=19.76%, ctx=2437, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4054KiB/s (4151kB/s), 1004KiB/s-1059KiB/s (1028kB/s-1084kB/s), io=23.4MiB (24.5MB), run=5705-5908msec WRITE: bw=4265KiB/s (4368kB/s), 1021KiB/s-1144KiB/s (1045kB/s-1172kB/s), io=24.6MiB (25.8MB), run=5705-5908msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18986: Wed Apr 17 12:53:49 2024 read: IOPS=66, BW=1065KiB/s (1091kB/s)(5920KiB/5557msec) clat (usec): min=1898, max=26860, avg=5761.56, stdev=4229.37 lat (usec): min=1899, max=26860, avg=5762.24, stdev=4229.35 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2278], 10.00th=[ 2474], 20.00th=[ 2737], | 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 3916], 60.00th=[ 5014], | 70.00th=[ 6128], 80.00th=[ 8848], 90.00th=[10945], 95.00th=[13042], | 99.00th=[23200], 99.50th=[24249], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 736, max= 1408, per=26.16%, avg=1036.30, stdev=213.49, samples=10 iops : min= 46, max= 88, avg=64.60, stdev=13.39, samples=10 write: IOPS=71, BW=1146KiB/s (1173kB/s)(6368KiB/5557msec) clat (msec): min=2, max=112, avg= 8.59, stdev= 7.10 lat (msec): min=2, max=112, avg= 8.59, stdev= 7.10 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 19], | 99.00th=[ 27], 99.50th=[ 29], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 864, max= 1280, per=27.16%, avg=1132.20, stdev=136.00, samples=10 iops : min= 54, max= 80, avg=70.60, stdev= 8.53, samples=10 lat (msec) : 2=0.26%, 4=28.78%, 10=48.57%, 20=20.18%, 50=2.08% lat (msec) : 250=0.13% cpu : usr=0.18%, sys=9.04%, ctx=917, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18987: Wed Apr 17 12:53:49 2024 read: IOPS=65, BW=1055KiB/s (1081kB/s)(5744KiB/5443msec) clat (msec): min=2, max=106, avg= 6.04, stdev= 6.70 lat (msec): min=2, max=106, avg= 6.04, stdev= 6.70 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 20], 99.50th=[ 21], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 864, max= 1408, per=26.57%, avg=1052.80, stdev=179.41, samples=10 iops : min= 54, max= 88, avg=65.80, stdev=11.21, samples=10 write: IOPS=75, BW=1202KiB/s (1231kB/s)(6544KiB/5443msec) clat (usec): min=2800, max=43630, avg=7992.30, stdev=4919.49 lat (usec): min=2801, max=43631, avg=7993.26, stdev=4919.50 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3720], 10.00th=[ 4047], 20.00th=[ 4359], | 30.00th=[ 4948], 40.00th=[ 5473], 50.00th=[ 6325], 60.00th=[ 7046], | 70.00th=[ 8717], 80.00th=[10552], 90.00th=[15270], 95.00th=[18744], | 99.00th=[23200], 99.50th=[25297], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 960, max= 1408, per=28.25%, avg=1177.60, stdev=125.12, samples=10 iops : min= 60, max= 88, avg=73.60, stdev= 7.82, samples=10 lat (msec) : 4=28.65%, 10=51.17%, 20=18.23%, 50=1.82%, 250=0.13% cpu : usr=0.09%, sys=9.30%, ctx=904, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18989: Wed Apr 17 12:53:49 2024 read: IOPS=64, BW=1035KiB/s (1060kB/s)(6256KiB/6045msec) clat (msec): min=2, max=108, avg= 6.16, stdev= 6.60 lat (msec): min=2, max=108, avg= 6.16, stdev= 6.60 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 25], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 640, max= 1312, per=26.02%, avg=1030.73, stdev=191.30, samples=11 iops : min= 40, max= 82, avg=64.36, stdev=11.89, samples=11 write: IOPS=62, BW=998KiB/s (1022kB/s)(6032KiB/6045msec) clat (usec): min=2862, max=41636, avg=9627.05, stdev=5110.00 lat (usec): min=2863, max=41637, avg=9628.43, stdev=5110.08 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3818], 10.00th=[ 4228], 20.00th=[ 4948], | 30.00th=[ 6259], 40.00th=[ 7504], 50.00th=[ 8848], 60.00th=[10028], | 70.00th=[11207], 80.00th=[13042], 90.00th=[16909], 95.00th=[19268], | 99.00th=[25035], 99.50th=[27919], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 672, max= 1312, per=23.59%, avg=983.09, stdev=200.10, samples=11 iops : min= 42, max= 82, avg=61.36, stdev=12.60, samples=11 lat (msec) : 4=27.60%, 10=44.40%, 20=24.87%, 50=2.99%, 250=0.13% cpu : usr=0.08%, sys=10.23%, ctx=1120, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18990: Wed Apr 17 12:53:49 2024 read: IOPS=70, BW=1128KiB/s (1155kB/s)(6032KiB/5348msec) clat (usec): min=2006, max=21454, avg=5613.12, stdev=3812.96 lat (usec): min=2007, max=21455, avg=5613.74, stdev=3813.01 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2343], 10.00th=[ 2474], 20.00th=[ 2802], | 30.00th=[ 3032], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4948], | 70.00th=[ 6390], 80.00th=[ 9110], 90.00th=[11469], 95.00th=[13698], | 99.00th=[18744], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 864, max= 1280, per=28.01%, avg=1109.90, stdev=177.83, samples=10 iops : min= 54, max= 80, avg=69.20, stdev=11.15, samples=10 write: IOPS=73, BW=1170KiB/s (1198kB/s)(6256KiB/5348msec) clat (usec): min=2977, max=60952, avg=8249.83, stdev=5553.38 lat (usec): min=2978, max=60953, avg=8250.77, stdev=5553.40 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3556], 10.00th=[ 3851], 20.00th=[ 4424], | 30.00th=[ 4883], 40.00th=[ 5604], 50.00th=[ 6456], 60.00th=[ 7308], | 70.00th=[ 8717], 80.00th=[11469], 90.00th=[15533], 95.00th=[18482], | 99.00th=[26608], 99.50th=[29492], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 990, max= 1376, per=28.01%, avg=1167.50, stdev=110.20, samples=10 iops : min= 61, max= 86, avg=72.80, stdev= 6.99, samples=10 lat (msec) : 4=32.03%, 10=47.79%, 20=18.10%, 50=1.95%, 100=0.13% cpu : usr=0.09%, sys=9.65%, ctx=883, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3962KiB/s (4057kB/s), 1035KiB/s-1128KiB/s (1060kB/s-1155kB/s), io=23.4MiB (24.5MB), run=5348-6045msec WRITE: bw=4169KiB/s (4269kB/s), 998KiB/s-1202KiB/s (1022kB/s-1231kB/s), io=24.6MiB (25.8MB), run=5348-6045msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19005: Wed Apr 17 12:53:52 2024 read: IOPS=14, BW=14.3MiB/s (14.9MB/s)(7168KiB/491msec) clat (usec): min=26646, max=59350, avg=38514.70, stdev=12353.86 lat (usec): min=26648, max=59350, avg=38515.94, stdev=12353.90 clat percentiles (usec): | 1.00th=[26608], 5.00th=[26608], 10.00th=[26608], 20.00th=[27395], | 30.00th=[32375], 40.00th=[32375], 50.00th=[34866], 60.00th=[36963], | 70.00th=[36963], 80.00th=[51643], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] write: IOPS=10, BW=10.2MiB/s (10.7MB/s)(5120KiB/491msec) clat (usec): min=22198, max=83833, avg=42581.77, stdev=25764.56 lat (usec): min=22245, max=83901, avg=42648.07, stdev=25761.72 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[22152], | 30.00th=[27132], 40.00th=[27132], 50.00th=[27919], 60.00th=[27919], | 70.00th=[51643], 80.00th=[51643], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] lat (msec) : 50=66.67%, 100=33.33% cpu : usr=0.00%, sys=26.53%, ctx=55, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19006: Wed Apr 17 12:53:52 2024 read: IOPS=1, BW=1253KiB/s (1283kB/s)(2048KiB/1635msec) clat (msec): min=49, max=780, avg=414.91, stdev=517.41 lat (msec): min=49, max=780, avg=414.91, stdev=517.41 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 785], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 1756, max= 1759, per=17.66%, avg=1757.50, stdev= 2.12, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6263KiB/s (6413kB/s)(10.0MiB/1635msec) clat (msec): min=21, max=408, avg=79.81, stdev=116.92 lat (msec): min=21, max=408, avg=79.88, stdev=116.91 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 26], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 71], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 1759, max= 8782, per=37.85%, avg=5273.00, stdev=3511.50, samples=3 iops : min= 1, max= 8, avg= 4.67, stdev= 3.51, samples=3 lat (msec) : 50=58.33%, 100=25.00%, 500=8.33%, 1000=8.33% cpu : usr=0.18%, sys=9.79%, ctx=73, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19007: Wed Apr 17 12:53:52 2024 read: IOPS=2, BW=2985KiB/s (3057kB/s)(6144KiB/2058msec) clat (msec): min=17, max=502, avg=127.09, stdev=187.10 lat (msec): min=17, max=502, avg=127.09, stdev=187.10 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 61], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 1744, max= 3488, per=26.29%, avg=2616.00, stdev=1233.19, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2985KiB/s (3057kB/s)(6144KiB/2058msec) clat (msec): min=24, max=819, avg=213.98, stdev=306.24 lat (msec): min=24, max=819, avg=214.03, stdev=306.23 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 85], | 70.00th=[ 243], 80.00th=[ 243], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 3488, max= 4096, per=27.22%, avg=3792.00, stdev=429.92, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=8.80%, ctx=111, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19008: Wed Apr 17 12:53:52 2024 read: IOPS=12, BW=12.6MiB/s (13.2MB/s)(5120KiB/398msec) clat (usec): min=16867, max=49665, avg=31500.29, stdev=15835.50 lat (usec): min=16868, max=49666, avg=31501.40, stdev=15835.29 clat percentiles (usec): | 1.00th=[16909], 5.00th=[16909], 10.00th=[16909], 20.00th=[16909], | 30.00th=[20579], 40.00th=[20579], 50.00th=[22676], 60.00th=[22676], | 70.00th=[47449], 80.00th=[47449], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] write: IOPS=17, BW=17.6MiB/s (18.4MB/s)(7168KiB/398msec) clat (usec): min=20486, max=49999, avg=32789.47, stdev=11675.86 lat (usec): min=20553, max=50078, avg=32858.43, stdev=11688.46 clat percentiles (usec): | 1.00th=[20579], 5.00th=[20579], 10.00th=[20579], 20.00th=[22676], | 30.00th=[23987], 40.00th=[23987], 50.00th=[30802], 60.00th=[35390], | 70.00th=[35390], 80.00th=[46400], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] lat (msec) : 20=8.33%, 50=91.67% cpu : usr=0.50%, sys=28.46%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9951KiB/s (10.2MB/s), 1253KiB/s-14.3MiB/s (1283kB/s-14.9MB/s), io=20.0MiB (20.0MB), run=398-2058msec WRITE: bw=13.6MiB/s (14.3MB/s), 2985KiB/s-17.6MiB/s (3057kB/s-18.4MB/s), io=28.0MiB (29.4MB), run=398-2058msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19009: Wed Apr 17 12:53:52 2024 read: IOPS=2, BW=2728KiB/s (2793kB/s)(7168KiB/2628msec) clat (usec): min=518, max=727960, avg=291016.00, stdev=323208.78 lat (usec): min=519, max=727961, avg=291017.03, stdev=323208.93 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 519], 10.00th=[ 519], 20.00th=[ 20841], | 30.00th=[ 52167], 40.00th=[ 52167], 50.00th=[ 77071], 60.00th=[509608], | 70.00th=[509608], 80.00th=[650118], 90.00th=[725615], 95.00th=[725615], | 99.00th=[725615], 99.50th=[725615], 99.90th=[725615], 99.95th=[725615], | 99.99th=[725615] bw ( KiB/s): min= 1822, max= 3650, per=29.80%, avg=2322.20, stdev=748.59, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=1, BW=1948KiB/s (1995kB/s)(5120KiB/2628msec) clat (msec): min=26, max=448, avg=117.53, stdev=185.48 lat (msec): min=26, max=448, avg=117.57, stdev=185.47 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 3650, max= 5466, per=41.78%, avg=4558.00, stdev=1284.11, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=41.67%, 100=16.67%, 500=8.33%, 750=25.00% cpu : usr=0.00%, sys=18.16%, ctx=992, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19010: Wed Apr 17 12:53:52 2024 read: IOPS=1, BW=1330KiB/s (1362kB/s)(2048KiB/1540msec) clat (msec): min=47, max=534, avg=291.03, stdev=344.58 lat (msec): min=47, max=534, avg=291.03, stdev=344.58 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 2048, max= 2048, per=26.28%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6649KiB/s (6809kB/s)(10.0MiB/1540msec) clat (msec): min=28, max=549, avg=95.62, stdev=159.81 lat (msec): min=28, max=549, avg=95.69, stdev=159.81 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 37], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 47], | 70.00th=[ 55], 80.00th=[ 55], 90.00th=[ 58], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max=10240, per=56.32%, avg=6144.00, stdev=4096.00, samples=3 iops : min= 2, max= 10, avg= 6.00, stdev= 4.00, samples=3 lat (msec) : 50=58.33%, 100=25.00%, 750=16.67% cpu : usr=0.06%, sys=22.74%, ctx=349, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19011: Wed Apr 17 12:53:52 2024 read: IOPS=2, BW=2409KiB/s (2467kB/s)(6144KiB/2550msec) clat (usec): min=537, max=742710, avg=227736.40, stdev=313093.43 lat (usec): min=538, max=742711, avg=227737.33, stdev=313093.64 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 603], | 30.00th=[ 603], 40.00th=[ 44827], 50.00th=[ 44827], 60.00th=[ 87557], | 70.00th=[488637], 80.00th=[488637], 90.00th=[742392], 95.00th=[742392], | 99.00th=[742392], 99.50th=[742392], 99.90th=[742392], 99.95th=[742392], | 99.99th=[742392] bw ( KiB/s): min= 2048, max= 4096, per=43.81%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2409KiB/s (2467kB/s)(6144KiB/2550msec) clat (msec): min=25, max=869, avg=195.15, stdev=332.30 lat (msec): min=25, max=870, avg=195.20, stdev=332.32 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 59], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 2035, max= 4096, per=22.50%, avg=2455.00, stdev=917.36, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 lat (usec) : 750=16.67% lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=19.38%, ctx=643, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19012: Wed Apr 17 12:53:52 2024 read: IOPS=1, BW=2017KiB/s (2065kB/s)(5120KiB/2539msec) clat (msec): min=2, max=588, avg=269.53, stdev=264.94 lat (msec): min=2, max=588, avg=269.54, stdev=264.94 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 2043, max= 6144, per=52.53%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=2, BW=2823KiB/s (2891kB/s)(7168KiB/2539msec) clat (msec): min=20, max=830, avg=169.48, stdev=293.50 lat (msec): min=20, max=830, avg=169.53, stdev=293.48 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 24], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 71], 60.00th=[ 96], | 70.00th=[ 96], 80.00th=[ 108], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2048, max= 6131, per=32.82%, avg=3580.75, stdev=1955.16, samples=4 iops : min= 2, max= 5, avg= 3.25, stdev= 1.50, samples=4 lat (msec) : 4=8.33%, 50=25.00%, 100=25.00%, 250=16.67%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=19.07%, ctx=652, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7793KiB/s (7980kB/s), 1330KiB/s-2728KiB/s (1362kB/s-2793kB/s), io=20.0MiB (20.0MB), run=1540-2628msec WRITE: bw=10.7MiB/s (11.2MB/s), 1948KiB/s-6649KiB/s (1995kB/s-6809kB/s), io=28.0MiB (29.4MB), run=1540-2628msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19020: Wed Apr 17 12:53:55 2024 read: IOPS=1, BW=6653KiB/s (6813kB/s)(12.0MiB/1847msec) clat (msec): min=163, max=880, avg=608.12, stdev=388.12 lat (msec): min=163, max=880, avg=608.12, stdev=388.12 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 785], 50.00th=[ 785], 60.00th=[ 785], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 5618, max= 8192, per=83.03%, avg=6905.00, stdev=1820.09, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 1000=66.67% cpu : usr=0.00%, sys=6.50%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19022: Wed Apr 17 12:53:55 2024 read: IOPS=0, BW=2079KiB/s (2129kB/s)(4096KiB/1970msec) clat (nsec): min=106387k, max=106387k, avg=106387398.00, stdev= 0.00 lat (nsec): min=106390k, max=106390k, avg=106389916.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=98.51%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4158KiB/s (4258kB/s)(8192KiB/1970msec) clat (msec): min=562, max=1254, avg=908.45, stdev=489.95 lat (msec): min=562, max=1255, avg=908.72, stdev=489.96 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 1250], | 70.00th=[ 1250], 80.00th=[ 1250], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7656, max= 7656, per=46.03%, avg=7656.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.05%, sys=10.46%, ctx=87, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19024: Wed Apr 17 12:53:55 2024 write: IOPS=1, BW=6564KiB/s (6722kB/s)(12.0MiB/1872msec) clat (msec): min=203, max=1110, avg=623.06, stdev=457.28 lat (msec): min=203, max=1110, avg=623.24, stdev=457.29 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 5843, max= 8175, per=42.14%, avg=7009.00, stdev=1648.97, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=7.38%, ctx=16, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19026: Wed Apr 17 12:53:55 2024 write: IOPS=5, BW=20.1MiB/s (21.1MB/s)(12.0MiB/597msec) clat (msec): min=86, max=327, avg=182.59, stdev=127.26 lat (msec): min=87, max=327, avg=182.82, stdev=127.18 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min=16384, max=16384, per=98.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=26.51%, ctx=84, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8317KiB/s (8516kB/s), 2079KiB/s-6653KiB/s (2129kB/s-6813kB/s), io=16.0MiB (16.8MB), run=1847-1970msec WRITE: bw=16.2MiB/s (17.0MB/s), 4158KiB/s-20.1MiB/s (4258kB/s-21.1MB/s), io=32.0MiB (33.6MB), run=597-1970msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19021: Wed Apr 17 12:53:55 2024 read: IOPS=1, BW=7161KiB/s (7333kB/s)(12.0MiB/1716msec) clat (msec): min=125, max=890, avg=571.17, stdev=398.19 lat (msec): min=125, max=890, avg=571.17, stdev=398.19 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 701], 50.00th=[ 701], 60.00th=[ 701], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 8192, max= 8192, per=85.81%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=65.77%, ctx=465, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19023: Wed Apr 17 12:53:55 2024 read: IOPS=0, BW=2484KiB/s (2544kB/s)(4096KiB/1649msec) clat (nsec): min=132260k, max=132260k, avg=132260219.00, stdev= 0.00 lat (nsec): min=132261k, max=132261k, avg=132261217.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 8192, max= 8192, per=85.81%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4968KiB/s (5087kB/s)(8192KiB/1649msec) clat (msec): min=437, max=1072, avg=755.09, stdev=449.09 lat (msec): min=437, max=1072, avg=755.33, stdev=449.12 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 1070], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8192, max= 8192, per=49.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=16.50%, ctx=118, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19025: Wed Apr 17 12:53:55 2024 write: IOPS=1, BW=6219KiB/s (6368kB/s)(12.0MiB/1976msec) clat (msec): min=398, max=1140, avg=656.21, stdev=419.48 lat (msec): min=398, max=1140, avg=656.50, stdev=419.46 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8192, max= 8192, per=49.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=14.94%, ctx=20, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19027: Wed Apr 17 12:53:55 2024 write: IOPS=1, BW=7666KiB/s (7850kB/s)(12.0MiB/1603msec) clat (msec): min=170, max=1170, avg=532.64, stdev=554.42 lat (msec): min=170, max=1171, avg=532.86, stdev=554.37 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8175, max= 8192, per=49.35%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=18.91%, ctx=42, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9548KiB/s (9777kB/s), 2484KiB/s-7161KiB/s (2544kB/s-7333kB/s), io=16.0MiB (16.8MB), run=1649-1716msec WRITE: bw=16.2MiB/s (16.0MB/s), 4968KiB/s-7666KiB/s (5087kB/s-7850kB/s), io=32.0MiB (33.6MB), run=1603-1976msec