== sanity test 398b: DIO and buffer IO race ============== 23:44:15 (1713411855) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.12235 s, 44.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15871: Wed Apr 17 23:44:30 2024 read: IOPS=129, BW=520KiB/s (532kB/s)(5992KiB/11528msec) clat (usec): min=56, max=162997, avg=4401.63, stdev=4739.32 lat (usec): min=57, max=162997, avg=4402.38, stdev=4739.31 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1713], 10.00th=[ 2089], 20.00th=[ 2474], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 3884], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 5997], 90.00th=[ 7177], 95.00th=[ 8291], | 99.00th=[ 12256], 99.50th=[ 13435], 99.90th=[ 29230], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 256, max= 694, per=25.26%, avg=519.57, stdev=80.65, samples=23 iops : min= 64, max= 173, avg=129.87, stdev=20.11, samples=23 write: IOPS=136, BW=546KiB/s (559kB/s)(6296KiB/11528msec) clat (usec): min=421, max=23888, avg=3088.78, stdev=2771.37 lat (usec): min=421, max=23889, avg=3089.46, stdev=2771.36 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 709], 40.00th=[ 1778], 50.00th=[ 2540], 60.00th=[ 3163], | 70.00th=[ 3982], 80.00th=[ 5080], 90.00th=[ 6915], 95.00th=[ 8455], | 99.00th=[11731], 99.50th=[13173], 99.90th=[19530], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 304, max= 702, per=25.33%, avg=545.30, stdev=82.25, samples=23 iops : min= 76, max= 175, avg=136.30, stdev=20.52, samples=23 lat (usec) : 100=1.79%, 250=0.26%, 500=1.14%, 750=15.36%, 1000=1.95% lat (msec) : 2=4.92%, 4=35.81%, 10=36.56%, 20=2.12%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.68%, ctx=5888, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15873: Wed Apr 17 23:44:30 2024 read: IOPS=128, BW=512KiB/s (525kB/s)(5940KiB/11591msec) clat (usec): min=49, max=19590, avg=4230.55, stdev=2373.69 lat (usec): min=50, max=19591, avg=4231.22, stdev=2373.72 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 1729], 10.00th=[ 2057], 20.00th=[ 2442], | 30.00th=[ 2835], 40.00th=[ 3261], 50.00th=[ 3785], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 5800], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[12649], 99.50th=[14091], 99.90th=[19006], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 256, max= 656, per=24.77%, avg=509.61, stdev=80.13, samples=23 iops : min= 64, max= 164, avg=127.30, stdev=20.01, samples=23 write: IOPS=136, BW=548KiB/s (561kB/s)(6348KiB/11591msec) clat (usec): min=414, max=158858, avg=3298.86, stdev=4814.97 lat (usec): min=415, max=158859, avg=3299.74, stdev=4815.02 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 603], | 30.00th=[ 816], 40.00th=[ 1975], 50.00th=[ 2704], 60.00th=[ 3294], | 70.00th=[ 4113], 80.00th=[ 5276], 90.00th=[ 6980], 95.00th=[ 8586], | 99.00th=[ 11731], 99.50th=[ 14091], 99.90th=[ 29230], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 336, max= 696, per=25.19%, avg=542.26, stdev=75.88, samples=23 iops : min= 84, max= 174, avg=135.48, stdev=18.91, samples=23 lat (usec) : 50=0.03%, 100=1.89%, 250=0.10%, 500=1.17%, 750=13.38% lat (usec) : 1000=2.70% lat (msec) : 2=5.83%, 4=36.17%, 10=36.10%, 20=2.57%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.72%, ctx=6102, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15875: Wed Apr 17 23:44:30 2024 read: IOPS=129, BW=516KiB/s (529kB/s)(6028KiB/11672msec) clat (usec): min=46, max=161399, avg=4311.30, stdev=4743.74 lat (usec): min=47, max=161400, avg=4312.10, stdev=4743.71 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 519], 10.00th=[ 2040], 20.00th=[ 2442], | 30.00th=[ 2835], 40.00th=[ 3228], 50.00th=[ 3752], 60.00th=[ 4424], | 70.00th=[ 5014], 80.00th=[ 5735], 90.00th=[ 7177], 95.00th=[ 8225], | 99.00th=[ 13042], 99.50th=[ 15008], 99.90th=[ 30540], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 280, max= 624, per=24.86%, avg=511.43, stdev=74.41, samples=23 iops : min= 70, max= 156, avg=127.78, stdev=18.60, samples=23 write: IOPS=134, BW=536KiB/s (549kB/s)(6260KiB/11672msec) clat (usec): min=404, max=19548, avg=3259.27, stdev=2873.90 lat (usec): min=404, max=19549, avg=3260.42, stdev=2873.53 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 758], 40.00th=[ 1926], 50.00th=[ 2638], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[11469], 99.50th=[12780], 99.90th=[17695], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 336, max= 718, per=24.64%, avg=530.57, stdev=88.74, samples=23 iops : min= 84, max= 179, avg=132.57, stdev=22.16, samples=23 lat (usec) : 50=0.13%, 100=2.12%, 250=0.10%, 500=1.33%, 750=13.93% lat (usec) : 1000=2.44% lat (msec) : 2=5.27%, 4=35.74%, 10=36.20%, 20=2.64%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=22.37%, ctx=6193, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15877: Wed Apr 17 23:44:30 2024 read: IOPS=129, BW=519KiB/s (531kB/s)(6052KiB/11662msec) clat (usec): min=47, max=17832, avg=4267.81, stdev=2351.01 lat (usec): min=48, max=17833, avg=4268.71, stdev=2350.94 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 906], 10.00th=[ 2073], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3884], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5932], 90.00th=[ 7111], 95.00th=[ 8291], | 99.00th=[12387], 99.50th=[13829], 99.90th=[15401], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 200, max= 648, per=25.01%, avg=514.48, stdev=90.05, samples=23 iops : min= 50, max= 162, avg=128.52, stdev=22.49, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6236KiB/11662msec) clat (usec): min=433, max=163571, avg=3290.92, stdev=4954.10 lat (usec): min=434, max=163573, avg=3291.57, stdev=4954.13 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 734], 40.00th=[ 1958], 50.00th=[ 2638], 60.00th=[ 3228], | 70.00th=[ 4080], 80.00th=[ 5407], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[ 12125], 99.50th=[ 14222], 99.90th=[ 31589], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 216, max= 744, per=24.64%, avg=530.48, stdev=106.64, samples=23 iops : min= 54, max= 186, avg=132.52, stdev=26.64, samples=23 lat (usec) : 50=0.10%, 100=2.08%, 250=0.20%, 500=0.98%, 750=14.81% lat (usec) : 1000=1.63% lat (msec) : 2=4.85%, 4=36.00%, 10=36.91%, 20=2.38%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.39%, ctx=6003, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2057KiB/s (2107kB/s), 512KiB/s-520KiB/s (525kB/s-532kB/s), io=23.4MiB (24.6MB), run=11528-11672msec WRITE: bw=2154KiB/s (2206kB/s), 535KiB/s-548KiB/s (548kB/s-561kB/s), io=24.6MiB (25.7MB), run=11528-11672msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15872: Wed Apr 17 23:44:34 2024 read: IOPS=92, BW=371KiB/s (380kB/s)(5992KiB/16160msec) clat (usec): min=1607, max=192689, avg=4388.06, stdev=5584.09 lat (usec): min=1608, max=192691, avg=4389.05, stdev=5584.13 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1926], 10.00th=[ 2057], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2802], 50.00th=[ 3326], 60.00th=[ 3916], | 70.00th=[ 4359], 80.00th=[ 5538], 90.00th=[ 8356], 95.00th=[ 10683], | 99.00th=[ 13698], 99.50th=[ 16057], 99.90th=[ 19530], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 176, max= 744, per=24.99%, avg=370.31, stdev=125.65, samples=32 iops : min= 44, max= 186, avg=92.47, stdev=31.45, samples=32 write: IOPS=97, BW=390KiB/s (399kB/s)(6296KiB/16160msec) clat (usec): min=2195, max=24623, avg=6044.83, stdev=3168.84 lat (usec): min=2196, max=24624, avg=6045.80, stdev=3168.82 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3458], | 30.00th=[ 3851], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 6194], | 70.00th=[ 6915], 80.00th=[ 7963], 90.00th=[10290], 95.00th=[12387], | 99.00th=[17433], 99.50th=[18744], 99.90th=[21890], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 192, max= 760, per=25.00%, avg=388.06, stdev=127.79, samples=32 iops : min= 48, max= 190, avg=96.91, stdev=31.96, samples=32 lat (msec) : 2=3.74%, 4=43.29%, 10=44.40%, 20=8.33%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.61%, ctx=3609, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15874: Wed Apr 17 23:44:34 2024 read: IOPS=92, BW=371KiB/s (379kB/s)(5940KiB/16031msec) clat (usec): min=1570, max=161276, avg=4312.38, stdev=4935.64 lat (usec): min=1571, max=161277, avg=4313.29, stdev=4935.65 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1893], 10.00th=[ 2057], 20.00th=[ 2311], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3294], 60.00th=[ 3851], | 70.00th=[ 4293], 80.00th=[ 5276], 90.00th=[ 7898], 95.00th=[ 10683], | 99.00th=[ 15008], 99.50th=[ 16909], 99.90th=[ 23987], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 168, max= 680, per=24.99%, avg=370.31, stdev=115.02, samples=32 iops : min= 42, max= 170, avg=92.47, stdev=28.73, samples=32 write: IOPS=98, BW=396KiB/s (405kB/s)(6348KiB/16031msec) clat (usec): min=2197, max=31586, avg=6020.41, stdev=3124.78 lat (usec): min=2197, max=31587, avg=6021.25, stdev=3124.72 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2900], 10.00th=[ 3163], 20.00th=[ 3556], | 30.00th=[ 3982], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 6063], | 70.00th=[ 6915], 80.00th=[ 7963], 90.00th=[10028], 95.00th=[11863], | 99.00th=[16909], 99.50th=[18482], 99.90th=[29754], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 224, max= 776, per=25.49%, avg=395.56, stdev=124.42, samples=32 iops : min= 56, max= 194, avg=98.78, stdev=31.11, samples=32 lat (msec) : 2=3.91%, 4=42.38%, 10=45.74%, 20=7.71%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.68%, ctx=3528, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15876: Wed Apr 17 23:44:34 2024 read: IOPS=93, BW=376KiB/s (385kB/s)(6028KiB/16050msec) clat (usec): min=1601, max=23962, avg=4249.69, stdev=2768.72 lat (usec): min=1602, max=23963, avg=4250.34, stdev=2768.71 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1893], 10.00th=[ 2040], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 3261], 60.00th=[ 3851], | 70.00th=[ 4359], 80.00th=[ 5604], 90.00th=[ 8291], 95.00th=[ 9896], | 99.00th=[14877], 99.50th=[16581], 99.90th=[20841], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 160, max= 728, per=25.31%, avg=375.06, stdev=127.23, samples=32 iops : min= 40, max= 182, avg=93.62, stdev=31.83, samples=32 write: IOPS=97, BW=390KiB/s (399kB/s)(6260KiB/16050msec) clat (msec): min=2, max=163, avg= 6.12, stdev= 5.08 lat (msec): min=2, max=163, avg= 6.12, stdev= 5.08 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 25], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 200, max= 776, per=25.03%, avg=388.53, stdev=130.11, samples=32 iops : min= 50, max= 194, avg=97.00, stdev=32.57, samples=32 lat (msec) : 2=4.00%, 4=43.65%, 10=44.86%, 20=7.16%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=11.46%, ctx=3623, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15878: Wed Apr 17 23:44:34 2024 read: IOPS=93, BW=374KiB/s (383kB/s)(6052KiB/16196msec) clat (usec): min=1519, max=21399, avg=4284.53, stdev=2860.55 lat (usec): min=1519, max=21400, avg=4285.19, stdev=2860.53 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2278], | 30.00th=[ 2507], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3851], | 70.00th=[ 4359], 80.00th=[ 5604], 90.00th=[ 8586], 95.00th=[10683], | 99.00th=[14222], 99.50th=[15401], 99.90th=[19006], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 176, max= 664, per=25.04%, avg=371.06, stdev=118.99, samples=32 iops : min= 44, max= 166, avg=92.66, stdev=29.78, samples=32 write: IOPS=96, BW=385KiB/s (394kB/s)(6236KiB/16196msec) clat (msec): min=2, max=163, avg= 6.18, stdev= 5.08 lat (msec): min=2, max=163, avg= 6.19, stdev= 5.08 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 17], 99.50th=[ 21], 99.90th=[ 31], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 200, max= 808, per=24.73%, avg=383.81, stdev=132.91, samples=32 iops : min= 50, max= 202, avg=95.84, stdev=33.24, samples=32 lat (msec) : 2=4.04%, 4=43.39%, 10=43.98%, 20=8.30%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.68%, ctx=3583, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1483KiB/s (1518kB/s), 371KiB/s-376KiB/s (379kB/s-385kB/s), io=23.4MiB (24.6MB), run=16031-16196msec WRITE: bw=1552KiB/s (1589kB/s), 385KiB/s-396KiB/s (394kB/s-405kB/s), io=24.6MiB (25.7MB), run=16031-16196msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15911: Wed Apr 17 23:44:41 2024 read: IOPS=60, BW=966KiB/s (989kB/s)(5920KiB/6127msec) clat (usec): min=69, max=48386, avg=10923.93, stdev=5145.46 lat (usec): min=70, max=48387, avg=10924.49, stdev=5145.50 clat percentiles (usec): | 1.00th=[ 133], 5.00th=[ 6521], 10.00th=[ 7373], 20.00th=[ 8291], | 30.00th=[ 9110], 40.00th=[ 9634], 50.00th=[10159], 60.00th=[10552], | 70.00th=[11338], 80.00th=[11994], 90.00th=[14484], 95.00th=[17171], | 99.00th=[36963], 99.50th=[47449], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 736, max= 1152, per=25.01%, avg=960.00, stdev=151.33, samples=12 iops : min= 46, max= 72, avg=60.00, stdev= 9.46, samples=12 write: IOPS=64, BW=1039KiB/s (1064kB/s)(6368KiB/6127msec) clat (usec): min=708, max=54233, avg=5222.00, stdev=5706.47 lat (usec): min=708, max=54234, avg=5222.93, stdev=5706.45 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 799], 10.00th=[ 865], 20.00th=[ 1991], | 30.00th=[ 2311], 40.00th=[ 2835], 50.00th=[ 3195], 60.00th=[ 4146], | 70.00th=[ 5473], 80.00th=[ 7504], 90.00th=[11863], 95.00th=[16581], | 99.00th=[31589], 99.50th=[40109], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 768, max= 1408, per=25.81%, avg=1042.67, stdev=215.08, samples=12 iops : min= 48, max= 88, avg=65.17, stdev=13.44, samples=12 lat (usec) : 100=0.39%, 250=0.26%, 750=0.39%, 1000=6.51% lat (msec) : 2=3.78%, 4=20.05%, 10=36.59%, 20=29.17%, 50=2.73% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=19.80%, ctx=2730, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15912: Wed Apr 17 23:44:41 2024 read: IOPS=57, BW=921KiB/s (943kB/s)(5744KiB/6238msec) clat (usec): min=59, max=51113, avg=10717.63, stdev=4126.58 lat (usec): min=59, max=51114, avg=10718.23, stdev=4126.57 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 6652], 10.00th=[ 7439], 20.00th=[ 8356], | 30.00th=[ 9110], 40.00th=[ 9765], 50.00th=[10421], 60.00th=[10814], | 70.00th=[11469], 80.00th=[12518], 90.00th=[14222], 95.00th=[16581], | 99.00th=[23987], 99.50th=[30016], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 480, max= 1149, per=23.61%, avg=906.25, stdev=193.79, samples=12 iops : min= 30, max= 71, avg=56.50, stdev=12.04, samples=12 write: IOPS=65, BW=1049KiB/s (1074kB/s)(6544KiB/6238msec) clat (usec): min=738, max=77069, avg=5827.42, stdev=7952.00 lat (usec): min=739, max=77070, avg=5828.38, stdev=7952.01 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 791], 10.00th=[ 840], 20.00th=[ 1336], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3261], 60.00th=[ 4178], | 70.00th=[ 5276], 80.00th=[ 7308], 90.00th=[13698], 95.00th=[18482], | 99.00th=[43254], 99.50th=[45351], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 768, max= 1408, per=25.21%, avg=1018.25, stdev=216.20, samples=12 iops : min= 48, max= 88, avg=63.50, stdev=13.56, samples=12 lat (usec) : 100=0.91%, 250=0.13%, 750=0.26%, 1000=8.59% lat (msec) : 2=3.12%, 4=19.01%, 10=33.59%, 20=30.73%, 50=3.39% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=18.41%, ctx=2730, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15913: Wed Apr 17 23:44:41 2024 read: IOPS=63, BW=1018KiB/s (1042kB/s)(6256KiB/6147msec) clat (usec): min=62, max=36541, avg=10665.08, stdev=3988.83 lat (usec): min=63, max=36542, avg=10666.75, stdev=3988.99 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 6849], 10.00th=[ 7570], 20.00th=[ 8455], | 30.00th=[ 8979], 40.00th=[ 9503], 50.00th=[10028], 60.00th=[10552], | 70.00th=[11207], 80.00th=[12256], 90.00th=[14091], 95.00th=[17433], | 99.00th=[26608], 99.50th=[29754], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 640, max= 1344, per=26.47%, avg=1016.00, stdev=208.34, samples=12 iops : min= 40, max= 84, avg=63.50, stdev=13.02, samples=12 write: IOPS=61, BW=981KiB/s (1005kB/s)(6032KiB/6147msec) clat (usec): min=687, max=74493, avg=5220.20, stdev=7437.90 lat (usec): min=688, max=74494, avg=5222.58, stdev=7438.22 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 1057], | 30.00th=[ 2147], 40.00th=[ 2474], 50.00th=[ 2966], 60.00th=[ 3785], | 70.00th=[ 4948], 80.00th=[ 6980], 90.00th=[11207], 95.00th=[16909], | 99.00th=[36439], 99.50th=[65274], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 640, max= 1344, per=23.83%, avg=962.67, stdev=218.51, samples=12 iops : min= 40, max= 84, avg=60.17, stdev=13.66, samples=12 lat (usec) : 100=0.52%, 250=0.39%, 750=0.52%, 1000=8.46% lat (msec) : 2=3.91%, 4=17.97%, 10=37.24%, 20=27.73%, 50=2.86% lat (msec) : 100=0.39% cpu : usr=0.07%, sys=17.80%, ctx=2752, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15914: Wed Apr 17 23:44:41 2024 read: IOPS=61, BW=992KiB/s (1016kB/s)(6032KiB/6082msec) clat (usec): min=64, max=57284, avg=10726.45, stdev=4601.55 lat (usec): min=64, max=57285, avg=10727.16, stdev=4601.65 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 6718], 10.00th=[ 7635], 20.00th=[ 8455], | 30.00th=[ 9110], 40.00th=[ 9634], 50.00th=[10028], 60.00th=[10814], | 70.00th=[11600], 80.00th=[12387], 90.00th=[14091], 95.00th=[16909], | 99.00th=[30278], 99.50th=[36439], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 608, max= 1280, per=25.63%, avg=984.00, stdev=178.49, samples=12 iops : min= 38, max= 80, avg=61.50, stdev=11.16, samples=12 write: IOPS=64, BW=1029KiB/s (1053kB/s)(6256KiB/6082msec) clat (usec): min=702, max=70298, avg=5196.24, stdev=6930.89 lat (usec): min=703, max=70300, avg=5197.21, stdev=6930.91 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 807], 10.00th=[ 840], 20.00th=[ 1303], | 30.00th=[ 2147], 40.00th=[ 2409], 50.00th=[ 3032], 60.00th=[ 3949], | 70.00th=[ 5211], 80.00th=[ 7111], 90.00th=[11207], 95.00th=[14746], | 99.00th=[43779], 99.50th=[52167], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 672, max= 1376, per=25.35%, avg=1024.00, stdev=197.73, samples=12 iops : min= 42, max= 86, avg=64.00, stdev=12.36, samples=12 lat (usec) : 100=1.30%, 250=0.13%, 750=0.52%, 1000=8.33% lat (msec) : 2=3.91%, 4=17.97%, 10=36.72%, 20=28.65%, 50=2.08% lat (msec) : 100=0.39% cpu : usr=0.05%, sys=19.36%, ctx=2729, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3840KiB/s (3932kB/s), 921KiB/s-1018KiB/s (943kB/s-1042kB/s), io=23.4MiB (24.5MB), run=6082-6238msec WRITE: bw=4040KiB/s (4137kB/s), 981KiB/s-1049KiB/s (1005kB/s-1074kB/s), io=24.6MiB (25.8MB), run=6082-6238msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15916: Wed Apr 17 23:44:41 2024 read: IOPS=60, BW=973KiB/s (997kB/s)(5920KiB/6083msec) clat (usec): min=1765, max=51153, avg=6138.00, stdev=5974.43 lat (usec): min=1765, max=51154, avg=6138.67, stdev=5974.36 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2474], | 30.00th=[ 2737], 40.00th=[ 3163], 50.00th=[ 3752], 60.00th=[ 4948], | 70.00th=[ 6783], 80.00th=[ 8848], 90.00th=[11994], 95.00th=[15795], | 99.00th=[33817], 99.50th=[47449], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 640, max= 1248, per=25.33%, avg=972.83, stdev=183.15, samples=12 iops : min= 40, max= 78, avg=60.67, stdev=11.41, samples=12 write: IOPS=65, BW=1047KiB/s (1072kB/s)(6368KiB/6083msec) clat (usec): min=2516, max=49917, avg=9560.87, stdev=6579.22 lat (usec): min=2516, max=49918, avg=9561.95, stdev=6579.24 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3228], 10.00th=[ 3818], 20.00th=[ 4752], | 30.00th=[ 5800], 40.00th=[ 6718], 50.00th=[ 7963], 60.00th=[ 8979], | 70.00th=[10552], 80.00th=[12911], 90.00th=[18482], 95.00th=[21365], | 99.00th=[42206], 99.50th=[44303], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 800, max= 1469, per=25.99%, avg=1050.17, stdev=210.80, samples=12 iops : min= 50, max= 91, avg=65.50, stdev=13.04, samples=12 lat (msec) : 2=0.52%, 4=31.38%, 10=43.49%, 20=20.31%, 50=4.17% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=9.36%, ctx=1179, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15917: Wed Apr 17 23:44:41 2024 read: IOPS=57, BW=921KiB/s (943kB/s)(5744KiB/6235msec) clat (usec): min=1758, max=53430, avg=6183.71, stdev=5384.60 lat (usec): min=1759, max=53430, avg=6184.32, stdev=5384.57 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2474], | 30.00th=[ 2737], 40.00th=[ 3130], 50.00th=[ 3982], 60.00th=[ 5604], | 70.00th=[ 7832], 80.00th=[ 9896], 90.00th=[11994], 95.00th=[14091], | 99.00th=[25560], 99.50th=[39584], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 608, max= 1312, per=23.88%, avg=917.33, stdev=215.46, samples=12 iops : min= 38, max= 82, avg=57.33, stdev=13.47, samples=12 write: IOPS=65, BW=1050KiB/s (1075kB/s)(6544KiB/6235msec) clat (usec): min=2384, max=70949, avg=9799.75, stdev=7452.74 lat (usec): min=2386, max=70950, avg=9800.88, stdev=7452.80 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3228], 10.00th=[ 3589], 20.00th=[ 4948], | 30.00th=[ 5669], 40.00th=[ 6783], 50.00th=[ 8160], 60.00th=[ 9110], | 70.00th=[11338], 80.00th=[13829], 90.00th=[16581], 95.00th=[21365], | 99.00th=[47973], 99.50th=[51119], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 832, max= 1344, per=25.34%, avg=1024.00, stdev=189.07, samples=12 iops : min= 52, max= 84, avg=64.00, stdev=11.82, samples=12 lat (msec) : 2=1.56%, 4=29.43%, 10=41.41%, 20=23.70%, 50=3.39% lat (msec) : 100=0.52% cpu : usr=0.10%, sys=9.21%, ctx=1240, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15918: Wed Apr 17 23:44:41 2024 read: IOPS=65, BW=1041KiB/s (1066kB/s)(6256KiB/6008msec) clat (usec): min=1770, max=47270, avg=6297.25, stdev=5660.02 lat (usec): min=1770, max=47271, avg=6297.80, stdev=5660.03 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2474], | 30.00th=[ 2737], 40.00th=[ 3359], 50.00th=[ 4146], 60.00th=[ 5342], | 70.00th=[ 7373], 80.00th=[ 9372], 90.00th=[12649], 95.00th=[16712], | 99.00th=[32113], 99.50th=[46400], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 702, max= 1373, per=27.12%, avg=1041.75, stdev=203.58, samples=12 iops : min= 43, max= 85, avg=64.83, stdev=12.68, samples=12 write: IOPS=62, BW=1004KiB/s (1028kB/s)(6032KiB/6008msec) clat (usec): min=2732, max=76409, avg=9386.41, stdev=7073.77 lat (usec): min=2733, max=76410, avg=9387.62, stdev=7073.76 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3228], 10.00th=[ 3556], 20.00th=[ 4293], | 30.00th=[ 5276], 40.00th=[ 6456], 50.00th=[ 7832], 60.00th=[ 9110], | 70.00th=[10290], 80.00th=[12911], 90.00th=[17171], 95.00th=[21890], | 99.00th=[35390], 99.50th=[50070], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 576, max= 1277, per=24.72%, avg=999.08, stdev=221.07, samples=12 iops : min= 36, max= 79, avg=62.17, stdev=13.66, samples=12 lat (msec) : 2=1.30%, 4=32.16%, 10=42.06%, 20=20.18%, 50=4.04% lat (msec) : 100=0.26% cpu : usr=0.05%, sys=9.44%, ctx=1192, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15919: Wed Apr 17 23:44:41 2024 read: IOPS=62, BW=1001KiB/s (1025kB/s)(6032KiB/6026msec) clat (usec): min=1846, max=54846, avg=6501.25, stdev=5703.27 lat (usec): min=1846, max=54847, avg=6501.82, stdev=5703.28 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3458], 50.00th=[ 4359], 60.00th=[ 6128], | 70.00th=[ 8094], 80.00th=[ 9765], 90.00th=[12649], 95.00th=[15270], | 99.00th=[33162], 99.50th=[36963], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 704, max= 1216, per=25.76%, avg=989.33, stdev=144.05, samples=12 iops : min= 44, max= 76, avg=61.83, stdev= 9.00, samples=12 write: IOPS=64, BW=1038KiB/s (1063kB/s)(6256KiB/6026msec) clat (usec): min=2536, max=48135, avg=9128.41, stdev=5921.04 lat (usec): min=2537, max=48136, avg=9129.51, stdev=5921.08 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3195], 10.00th=[ 3556], 20.00th=[ 4752], | 30.00th=[ 5735], 40.00th=[ 6652], 50.00th=[ 7701], 60.00th=[ 8848], | 70.00th=[10159], 80.00th=[12125], 90.00th=[16057], 95.00th=[21103], | 99.00th=[33162], 99.50th=[36439], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 800, max= 1344, per=25.67%, avg=1037.33, stdev=171.76, samples=12 iops : min= 50, max= 84, avg=64.83, stdev=10.74, samples=12 lat (msec) : 2=1.30%, 4=28.65%, 10=44.92%, 20=20.96%, 50=4.04% lat (msec) : 100=0.13% cpu : usr=0.03%, sys=9.51%, ctx=1224, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3842KiB/s (3934kB/s), 921KiB/s-1041KiB/s (943kB/s-1066kB/s), io=23.4MiB (24.5MB), run=6008-6235msec WRITE: bw=4042KiB/s (4139kB/s), 1004KiB/s-1050KiB/s (1028kB/s-1075kB/s), io=24.6MiB (25.8MB), run=6008-6235msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15930: Wed Apr 17 23:44:44 2024 read: IOPS=5, BW=5255KiB/s (5381kB/s)(7168KiB/1364msec) clat (usec): min=320, max=837857, avg=171365.92, stdev=307929.77 lat (usec): min=320, max=837858, avg=171366.82, stdev=307929.63 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 322], 10.00th=[ 322], 20.00th=[ 326], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 433], 60.00th=[119014], | 70.00th=[119014], 80.00th=[242222], 90.00th=[834667], 95.00th=[834667], | 99.00th=[834667], 99.50th=[834667], 99.90th=[834667], 99.95th=[834667], | 99.99th=[834667] bw ( KiB/s): min= 1768, max=10219, per=39.92%, avg=5993.50, stdev=5975.76, samples=2 iops : min= 1, max= 9, avg= 5.00, stdev= 5.66, samples=2 write: IOPS=3, BW=3754KiB/s (3844kB/s)(5120KiB/1364msec) clat (usec): min=18665, max=60634, avg=32086.35, stdev=18111.78 lat (usec): min=18683, max=60707, avg=32114.57, stdev=18134.51 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[18744], | 30.00th=[18744], 40.00th=[18744], 50.00th=[22938], 60.00th=[22938], | 70.00th=[39584], 80.00th=[39584], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 3537, max= 6131, per=23.00%, avg=4834.00, stdev=1834.23, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (usec) : 500=33.33% lat (msec) : 20=16.67%, 50=16.67%, 100=8.33%, 250=16.67%, 1000=8.33% cpu : usr=0.00%, sys=27.95%, ctx=404, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15931: Wed Apr 17 23:44:44 2024 read: IOPS=4, BW=4249KiB/s (4351kB/s)(2048KiB/482msec) clat (usec): min=463, max=131542, avg=66002.84, stdev=92686.38 lat (usec): min=464, max=131544, avg=66004.58, stdev=92687.85 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 465], | 30.00th=[ 465], 40.00th=[ 465], 50.00th=[ 465], 60.00th=[131597], | 70.00th=[131597], 80.00th=[131597], 90.00th=[131597], 95.00th=[131597], | 99.00th=[131597], 99.50th=[131597], 99.90th=[131597], 99.95th=[131597], | 99.99th=[131597] write: IOPS=20, BW=20.7MiB/s (21.8MB/s)(10.0MiB/482msec) clat (usec): min=18395, max=59448, avg=34703.15, stdev=13326.98 lat (usec): min=18435, max=59513, avg=34756.66, stdev=13324.13 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[23462], | 30.00th=[27132], 40.00th=[27919], 50.00th=[30016], 60.00th=[30540], | 70.00th=[46924], 80.00th=[46924], 90.00th=[51643], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=56.76%, ctx=112, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15932: Wed Apr 17 23:44:44 2024 read: IOPS=16, BW=16.6MiB/s (17.4MB/s)(6144KiB/362msec) clat (usec): min=327, max=55803, avg=17109.86, stdev=24602.47 lat (usec): min=327, max=55804, avg=17110.54, stdev=24602.48 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 429], | 30.00th=[ 429], 40.00th=[ 1614], 50.00th=[ 1614], 60.00th=[ 3851], | 70.00th=[40633], 80.00th=[40633], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] write: IOPS=16, BW=16.6MiB/s (17.4MB/s)(6144KiB/362msec) clat (usec): min=24423, max=81554, avg=42470.28, stdev=20647.22 lat (usec): min=24441, max=81590, avg=42512.19, stdev=20649.44 clat percentiles (usec): | 1.00th=[24511], 5.00th=[24511], 10.00th=[24511], 20.00th=[31589], | 30.00th=[31589], 40.00th=[32113], 50.00th=[32113], 60.00th=[37487], | 70.00th=[47973], 80.00th=[47973], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] lat (usec) : 500=16.67% lat (msec) : 2=8.33%, 4=8.33%, 50=50.00%, 100=16.67% cpu : usr=0.00%, sys=46.54%, ctx=68, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15933: Wed Apr 17 23:44:44 2024 read: IOPS=11, BW=11.3MiB/s (11.9MB/s)(5120KiB/442msec) clat (usec): min=335, max=96677, avg=26679.03, stdev=40615.40 lat (usec): min=335, max=96678, avg=26679.64, stdev=40615.40 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 392], 40.00th=[ 392], 50.00th=[ 8979], 60.00th=[ 8979], | 70.00th=[27132], 80.00th=[27132], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] write: IOPS=15, BW=15.8MiB/s (16.6MB/s)(7168KiB/442msec) clat (usec): min=23227, max=72559, avg=43520.24, stdev=22118.30 lat (usec): min=23286, max=72581, avg=43558.10, stdev=22104.20 clat percentiles (usec): | 1.00th=[23200], 5.00th=[23200], 10.00th=[23200], 20.00th=[23725], | 30.00th=[26346], 40.00th=[26346], 50.00th=[31065], 60.00th=[63701], | 70.00th=[63701], 80.00th=[64226], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 50=41.67%, 100=33.33% cpu : usr=0.00%, sys=42.18%, ctx=156, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.7MiB/s (15.4MB/s), 4249KiB/s-16.6MiB/s (4351kB/s-17.4MB/s), io=20.0MiB (20.0MB), run=362-1364msec WRITE: bw=20.5MiB/s (21.5MB/s), 3754KiB/s-20.7MiB/s (3844kB/s-21.8MB/s), io=28.0MiB (29.4MB), run=362-1364msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15934: Wed Apr 17 23:44:44 2024 read: IOPS=4, BW=5091KiB/s (5213kB/s)(7168KiB/1408msec) clat (msec): min=17, max=640, avg=170.88, stdev=226.07 lat (msec): min=17, max=640, avg=170.89, stdev=226.07 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 84], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 288], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 5657, max= 6131, per=43.54%, avg=5894.00, stdev=335.17, samples=2 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3636KiB/s (3724kB/s)(5120KiB/1408msec) clat (usec): min=24761, max=55673, avg=36215.55, stdev=13792.84 lat (usec): min=24836, max=55740, avg=36272.27, stdev=13795.46 clat percentiles (usec): | 1.00th=[24773], 5.00th=[24773], 10.00th=[24773], 20.00th=[24773], | 30.00th=[25035], 40.00th=[25035], 50.00th=[30016], 60.00th=[30016], | 70.00th=[45351], 80.00th=[45351], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 2043, max= 7543, per=25.29%, avg=4793.00, stdev=3889.09, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=8.96%, ctx=60, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15935: Wed Apr 17 23:44:44 2024 read: IOPS=1, BW=1513KiB/s (1549kB/s)(2048KiB/1354msec) clat (usec): min=18182, max=91533, avg=54857.80, stdev=51866.70 lat (usec): min=18183, max=91534, avg=54858.92, stdev=51867.43 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[18220], | 30.00th=[18220], 40.00th=[18220], 50.00th=[18220], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 4087, max= 4087, per=30.19%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7563KiB/s (7744kB/s)(10.0MiB/1354msec) clat (msec): min=19, max=740, avg=124.27, stdev=218.82 lat (msec): min=19, max=740, avg=124.31, stdev=218.85 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 22], | 30.00th=[ 25], 40.00th=[ 41], 50.00th=[ 50], 60.00th=[ 61], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 105], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7460, max=10219, per=46.65%, avg=8839.50, stdev=1950.91, samples=2 iops : min= 7, max= 9, avg= 8.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=12.12%, ctx=80, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15936: Wed Apr 17 23:44:44 2024 read: IOPS=7, BW=7670KiB/s (7855kB/s)(6144KiB/801msec) clat (usec): min=18235, max=66107, avg=34590.72, stdev=18721.23 lat (usec): min=18235, max=66108, avg=34591.72, stdev=18721.41 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[21627], | 30.00th=[21627], 40.00th=[23725], 50.00th=[23725], 60.00th=[29754], | 70.00th=[47973], 80.00th=[47973], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 2048, max= 2048, per=15.13%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7670KiB/s (7855kB/s)(6144KiB/801msec) clat (msec): min=23, max=271, avg=90.41, stdev=90.40 lat (msec): min=23, max=271, avg=90.45, stdev=90.40 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 68], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 6144, max= 6144, per=32.42%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 500=8.33% cpu : usr=0.00%, sys=16.12%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15937: Wed Apr 17 23:44:44 2024 read: IOPS=3, BW=3384KiB/s (3465kB/s)(5120KiB/1513msec) clat (msec): min=24, max=766, avg=178.29, stdev=328.65 lat (msec): min=24, max=766, avg=178.29, stdev=328.65 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 33], 60.00th=[ 33], | 70.00th=[ 42], 80.00th=[ 42], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2048, max= 6144, per=30.26%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=4, BW=4738KiB/s (4851kB/s)(7168KiB/1513msec) clat (msec): min=28, max=286, avg=85.94, stdev=93.76 lat (msec): min=28, max=287, avg=85.99, stdev=93.76 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 37], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 99], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 2048, max= 6144, per=25.22%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (msec) : 50=66.67%, 100=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=8.20%, ctx=52, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.2MiB/s (13.9MB/s), 1513KiB/s-7670KiB/s (1549kB/s-7855kB/s), io=20.0MiB (20.0MB), run=801-1513msec WRITE: bw=18.5MiB/s (19.4MB/s), 3636KiB/s-7670KiB/s (3724kB/s-7855kB/s), io=28.0MiB (29.4MB), run=801-1513msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15943: Wed Apr 17 23:44:50 2024 read: IOPS=0, BW=2439KiB/s (2497kB/s)(12.0MiB/5039msec) clat (msec): min=115, max=3323, avg=1677.11, stdev=1605.84 lat (msec): min=115, max=3323, avg=1677.11, stdev=1605.84 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 1586], 50.00th=[ 1586], 60.00th=[ 1586], | 70.00th=[ 3339], 80.00th=[ 3339], 90.00th=[ 3339], 95.00th=[ 3339], | 99.00th=[ 3339], 99.50th=[ 3339], 99.90th=[ 3339], 99.95th=[ 3339], | 99.99th=[ 3339] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=35.11%, ctx=1310, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15944: Wed Apr 17 23:44:50 2024 read: IOPS=0, BW=721KiB/s (738kB/s)(4096KiB/5681msec) clat (nsec): min=141682k, max=141682k, avg=141682388.00, stdev= 0.00 lat (nsec): min=141684k, max=141684k, avg=141683804.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1442KiB/s (1477kB/s)(8192KiB/5681msec) clat (msec): min=730, max=4808, avg=2769.14, stdev=2883.52 lat (msec): min=730, max=4808, avg=2769.36, stdev=2883.69 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 735], 10.00th=[ 735], 20.00th=[ 735], | 30.00th=[ 735], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 4799], | 70.00th=[ 4799], 80.00th=[ 4799], 90.00th=[ 4799], 95.00th=[ 4799], | 99.00th=[ 4799], 99.50th=[ 4799], 99.90th=[ 4799], 99.95th=[ 4799], | 99.99th=[ 4799] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=5.49%, ctx=647, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15945: Wed Apr 17 23:44:50 2024 write: IOPS=0, BW=2324KiB/s (2380kB/s)(12.0MiB/5288msec) clat (msec): min=319, max=3141, avg=1760.92, stdev=1412.03 lat (msec): min=319, max=3141, avg=1761.13, stdev=1412.01 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 1821], 50.00th=[ 1821], 60.00th=[ 1821], | 70.00th=[ 3138], 80.00th=[ 3138], 90.00th=[ 3138], 95.00th=[ 3138], | 99.00th=[ 3138], 99.50th=[ 3138], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.02%, sys=8.47%, ctx=480, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15946: Wed Apr 17 23:44:50 2024 write: IOPS=0, BW=2291KiB/s (2346kB/s)(12.0MiB/5364msec) clat (msec): min=88, max=4673, avg=1782.04, stdev=2515.99 lat (msec): min=88, max=4673, avg=1782.30, stdev=2516.00 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 4665], 80.00th=[ 4665], 90.00th=[ 4665], 95.00th=[ 4665], | 99.00th=[ 4665], 99.50th=[ 4665], 99.90th=[ 4665], 99.95th=[ 4665], | 99.99th=[ 4665] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=5.31%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2884KiB/s (2953kB/s), 721KiB/s-2439KiB/s (738kB/s-2497kB/s), io=16.0MiB (16.8MB), run=5039-5681msec WRITE: bw=5768KiB/s (5906kB/s), 1442KiB/s-2324KiB/s (1477kB/s-2380kB/s), io=32.0MiB (33.6MB), run=5288-5681msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15948: Wed Apr 17 23:44:51 2024 read: IOPS=0, BW=2227KiB/s (2281kB/s)(12.0MiB/5517msec) clat (msec): min=52, max=4878, avg=1838.67, stdev=2645.90 lat (msec): min=52, max=4878, avg=1838.67, stdev=2645.90 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 4866], 80.00th=[ 4866], 90.00th=[ 4866], 95.00th=[ 4866], | 99.00th=[ 4866], 99.50th=[ 4866], 99.90th=[ 4866], 99.95th=[ 4866], | 99.99th=[ 4866] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=1.94%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15949: Wed Apr 17 23:44:51 2024 read: IOPS=0, BW=716KiB/s (733kB/s)(4096KiB/5722msec) clat (nsec): min=539794k, max=539794k, avg=539793658.00, stdev= 0.00 lat (nsec): min=539795k, max=539795k, avg=539795252.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1432KiB/s (1466kB/s)(8192KiB/5722msec) clat (msec): min=396, max=4774, avg=2585.38, stdev=3096.14 lat (msec): min=396, max=4774, avg=2585.61, stdev=3095.97 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 4799], | 70.00th=[ 4799], 80.00th=[ 4799], 90.00th=[ 4799], 95.00th=[ 4799], | 99.00th=[ 4799], 99.50th=[ 4799], 99.90th=[ 4799], 99.95th=[ 4799], | 99.99th=[ 4799] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.02%, sys=2.57%, ctx=175, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15950: Wed Apr 17 23:44:51 2024 write: IOPS=0, BW=2233KiB/s (2287kB/s)(12.0MiB/5502msec) clat (msec): min=502, max=3824, avg=1831.86, stdev=1757.51 lat (msec): min=502, max=3824, avg=1832.08, stdev=1757.59 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 1167], 50.00th=[ 1167], 60.00th=[ 1167], | 70.00th=[ 3809], 80.00th=[ 3809], 90.00th=[ 3809], 95.00th=[ 3809], | 99.00th=[ 3809], 99.50th=[ 3809], 99.90th=[ 3809], 99.95th=[ 3809], | 99.99th=[ 3809] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33% cpu : usr=0.02%, sys=4.07%, ctx=106, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15951: Wed Apr 17 23:44:51 2024 write: IOPS=0, BW=2196KiB/s (2249kB/s)(12.0MiB/5595msec) clat (msec): min=67, max=4849, avg=1860.51, stdev=2605.51 lat (msec): min=67, max=4849, avg=1860.77, stdev=2605.53 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 4866], 80.00th=[ 4866], 90.00th=[ 4866], 95.00th=[ 4866], | 99.00th=[ 4866], 99.50th=[ 4866], 99.90th=[ 4866], 99.95th=[ 4866], | 99.99th=[ 4866] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.04%, sys=2.41%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2863KiB/s (2932kB/s), 716KiB/s-2227KiB/s (733kB/s-2281kB/s), io=16.0MiB (16.8MB), run=5517-5722msec WRITE: bw=5727KiB/s (5864kB/s), 1432KiB/s-2233KiB/s (1466kB/s-2287kB/s), io=32.0MiB (33.6MB), run=5502-5722msec