== sanity test 398b: DIO and buffer IO race ============== 15:45:54 (1713296754) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18822 s, 42.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30252: Tue Apr 16 15:46:12 2024 read: IOPS=95, BW=384KiB/s (393kB/s)(5992KiB/15611msec) clat (usec): min=48, max=179303, avg=4643.01, stdev=6092.38 lat (usec): min=48, max=179304, avg=4643.97, stdev=6092.43 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 72], 10.00th=[ 91], 20.00th=[ 147], | 30.00th=[ 2638], 40.00th=[ 3097], 50.00th=[ 3687], 60.00th=[ 4490], | 70.00th=[ 5604], 80.00th=[ 7308], 90.00th=[ 9765], 95.00th=[ 11731], | 99.00th=[ 15664], 99.50th=[ 21365], 99.90th=[ 41157], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 208, max= 566, per=24.48%, avg=376.43, stdev=81.30, samples=30 iops : min= 52, max= 141, avg=94.07, stdev=20.31, samples=30 write: IOPS=100, BW=403KiB/s (413kB/s)(6296KiB/15611msec) clat (usec): min=437, max=68949, avg=5445.02, stdev=5332.03 lat (usec): min=438, max=68951, avg=5445.95, stdev=5331.99 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 578], 10.00th=[ 652], 20.00th=[ 865], | 30.00th=[ 1254], 40.00th=[ 2966], 50.00th=[ 4293], 60.00th=[ 5407], | 70.00th=[ 6849], 80.00th=[ 9110], 90.00th=[12387], 95.00th=[15270], | 99.00th=[22152], 99.50th=[26608], 99.90th=[41157], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 272, max= 496, per=24.53%, avg=394.87, stdev=65.08, samples=30 iops : min= 68, max= 124, avg=98.67, stdev=16.28, samples=30 lat (usec) : 50=0.13%, 100=5.76%, 250=4.33%, 500=0.42%, 750=7.36% lat (usec) : 1000=5.86% lat (msec) : 2=4.69%, 4=21.87%, 10=36.39%, 20=12.21%, 50=0.91% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=23.89%, ctx=6349, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30253: Tue Apr 16 15:46:12 2024 read: IOPS=96, BW=387KiB/s (396kB/s)(5940KiB/15350msec) clat (usec): min=48, max=41837, avg=4391.80, stdev=4064.41 lat (usec): min=49, max=41838, avg=4393.05, stdev=4064.66 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 74], 10.00th=[ 92], 20.00th=[ 133], | 30.00th=[ 2606], 40.00th=[ 3032], 50.00th=[ 3621], 60.00th=[ 4228], | 70.00th=[ 5276], 80.00th=[ 7242], 90.00th=[ 9765], 95.00th=[11731], | 99.00th=[16319], 99.50th=[21890], 99.90th=[34866], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 280, max= 552, per=25.12%, avg=386.33, stdev=80.36, samples=30 iops : min= 70, max= 138, avg=96.53, stdev=20.07, samples=30 write: IOPS=103, BW=414KiB/s (423kB/s)(6348KiB/15350msec) clat (usec): min=473, max=178264, avg=5508.71, stdev=6845.24 lat (usec): min=474, max=178265, avg=5509.69, stdev=6845.17 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 578], 10.00th=[ 701], 20.00th=[ 865], | 30.00th=[ 1156], 40.00th=[ 2573], 50.00th=[ 4228], 60.00th=[ 5407], | 70.00th=[ 6783], 80.00th=[ 9110], 90.00th=[ 12125], 95.00th=[ 15401], | 99.00th=[ 23725], 99.50th=[ 28443], 99.90th=[ 46400], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 288, max= 566, per=25.32%, avg=407.63, stdev=75.36, samples=30 iops : min= 72, max= 141, avg=101.87, stdev=18.83, samples=30 lat (usec) : 50=0.07%, 100=5.83%, 250=5.14%, 500=0.46%, 750=6.09% lat (usec) : 1000=7.62% lat (msec) : 2=5.18%, 4=22.20%, 10=34.47%, 20=11.62%, 50=1.30% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.90%, ctx=6189, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30254: Tue Apr 16 15:46:12 2024 read: IOPS=97, BW=390KiB/s (399kB/s)(6028KiB/15470msec) clat (usec): min=46, max=179606, avg=4433.00, stdev=5976.95 lat (usec): min=47, max=179607, avg=4434.09, stdev=5976.71 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 73], 10.00th=[ 89], 20.00th=[ 128], | 30.00th=[ 2573], 40.00th=[ 3130], 50.00th=[ 3654], 60.00th=[ 4293], | 70.00th=[ 5276], 80.00th=[ 6980], 90.00th=[ 9372], 95.00th=[ 11338], | 99.00th=[ 15139], 99.50th=[ 17957], 99.90th=[ 55313], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 264, max= 560, per=25.24%, avg=388.20, stdev=75.58, samples=30 iops : min= 66, max= 140, avg=97.00, stdev=18.91, samples=30 write: IOPS=101, BW=405KiB/s (414kB/s)(6260KiB/15470msec) clat (usec): min=455, max=60720, avg=5561.74, stdev=5698.01 lat (usec): min=455, max=60721, avg=5562.78, stdev=5697.87 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 586], 10.00th=[ 676], 20.00th=[ 881], | 30.00th=[ 1319], 40.00th=[ 2802], 50.00th=[ 4359], 60.00th=[ 5407], | 70.00th=[ 6980], 80.00th=[ 8979], 90.00th=[12649], 95.00th=[16057], | 99.00th=[24511], 99.50th=[30016], 99.90th=[54789], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 264, max= 544, per=24.94%, avg=401.53, stdev=65.01, samples=30 iops : min= 66, max= 136, avg=100.33, stdev=16.26, samples=30 lat (usec) : 50=0.23%, 100=5.86%, 250=5.24%, 500=0.72%, 750=6.09% lat (usec) : 1000=6.45% lat (msec) : 2=5.60%, 4=21.65%, 10=35.81%, 20=11.10%, 50=1.14% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.20%, sys=23.87%, ctx=6219, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30255: Tue Apr 16 15:46:12 2024 read: IOPS=98, BW=396KiB/s (405kB/s)(6052KiB/15302msec) clat (usec): min=48, max=37841, avg=4316.12, stdev=3870.87 lat (usec): min=48, max=37841, avg=4317.00, stdev=3870.79 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 72], 10.00th=[ 90], 20.00th=[ 128], | 30.00th=[ 2573], 40.00th=[ 3064], 50.00th=[ 3589], 60.00th=[ 4228], | 70.00th=[ 5276], 80.00th=[ 6849], 90.00th=[ 9372], 95.00th=[11469], | 99.00th=[15926], 99.50th=[19268], 99.90th=[35390], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 208, max= 536, per=25.57%, avg=393.23, stdev=76.61, samples=30 iops : min= 52, max= 134, avg=98.27, stdev=19.15, samples=30 write: IOPS=101, BW=408KiB/s (417kB/s)(6236KiB/15302msec) clat (usec): min=453, max=188517, avg=5571.59, stdev=7216.16 lat (usec): min=454, max=188519, avg=5573.26, stdev=7216.28 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 570], 10.00th=[ 660], 20.00th=[ 865], | 30.00th=[ 1254], 40.00th=[ 2802], 50.00th=[ 4293], 60.00th=[ 5473], | 70.00th=[ 6783], 80.00th=[ 8979], 90.00th=[ 12125], 95.00th=[ 15795], | 99.00th=[ 23987], 99.50th=[ 34341], 99.90th=[ 55313], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 224, max= 544, per=25.16%, avg=405.00, stdev=78.06, samples=30 iops : min= 56, max= 136, avg=101.20, stdev=19.56, samples=30 lat (usec) : 50=0.16%, 100=6.15%, 250=4.92%, 500=0.49%, 750=6.74% lat (usec) : 1000=6.38% lat (msec) : 2=5.57%, 4=22.40%, 10=34.67%, 20=11.23%, 50=1.24% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.22%, sys=23.97%, ctx=6151, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1538KiB/s (1575kB/s), 384KiB/s-396KiB/s (393kB/s-405kB/s), io=23.4MiB (24.6MB), run=15302-15611msec WRITE: bw=1610KiB/s (1649kB/s), 403KiB/s-414KiB/s (413kB/s-423kB/s), io=24.6MiB (25.7MB), run=15302-15611msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30257: Tue Apr 16 15:46:44 2024 read: IOPS=31, BW=126KiB/s (129kB/s)(5992KiB/47622msec) clat (usec): min=1816, max=181838, avg=4352.45, stdev=6385.28 lat (usec): min=1817, max=181840, avg=4353.63, stdev=6385.23 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2671], 50.00th=[ 2802], 60.00th=[ 2999], | 70.00th=[ 3326], 80.00th=[ 4293], 90.00th=[ 8717], 95.00th=[ 12125], | 99.00th=[ 20055], 99.50th=[ 28181], 99.90th=[ 73925], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 24, max= 248, per=25.16%, avg=126.07, stdev=51.11, samples=94 iops : min= 6, max= 62, avg=31.48, stdev=12.78, samples=94 write: IOPS=33, BW=132KiB/s (135kB/s)(6296KiB/47622msec) clat (usec): min=12856, max=67009, avg=26079.02, stdev=8924.59 lat (usec): min=12857, max=67010, avg=26080.24, stdev=8924.59 clat percentiles (usec): | 1.00th=[13829], 5.00th=[15401], 10.00th=[16581], 20.00th=[18482], | 30.00th=[20579], 40.00th=[22676], 50.00th=[24773], 60.00th=[26608], | 70.00th=[28181], 80.00th=[31065], 90.00th=[38011], 95.00th=[46400], | 99.00th=[54789], 99.50th=[58983], 99.90th=[63177], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 64, max= 176, per=25.21%, avg=132.12, stdev=35.63, samples=94 iops : min= 16, max= 44, avg=32.99, stdev= 8.92, samples=94 lat (msec) : 2=1.07%, 4=37.17%, 10=6.58%, 20=17.06%, 50=36.65% lat (msec) : 100=1.43%, 250=0.03% cpu : usr=0.12%, sys=4.76%, ctx=3340, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30258: Tue Apr 16 15:46:44 2024 read: IOPS=31, BW=124KiB/s (127kB/s)(5940KiB/47897msec) clat (usec): min=1639, max=173323, avg=4334.63, stdev=6203.26 lat (usec): min=1640, max=173325, avg=4335.56, stdev=6203.28 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 2999], | 70.00th=[ 3326], 80.00th=[ 4293], 90.00th=[ 8455], 95.00th=[ 11994], | 99.00th=[ 18220], 99.50th=[ 27657], 99.90th=[ 73925], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 16, max= 264, per=24.71%, avg=123.81, stdev=52.15, samples=95 iops : min= 4, max= 66, avg=30.93, stdev=13.04, samples=95 write: IOPS=33, BW=133KiB/s (136kB/s)(6348KiB/47897msec) clat (usec): min=12807, max=81773, avg=26102.39, stdev=9259.39 lat (usec): min=12808, max=81774, avg=26103.75, stdev=9259.40 clat percentiles (usec): | 1.00th=[13829], 5.00th=[15270], 10.00th=[16581], 20.00th=[18220], | 30.00th=[20317], 40.00th=[22676], 50.00th=[24511], 60.00th=[26346], | 70.00th=[28181], 80.00th=[31065], 90.00th=[39584], 95.00th=[44827], | 99.00th=[55837], 99.50th=[61080], 99.90th=[71828], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 63, max= 176, per=25.14%, avg=131.76, stdev=35.84, samples=95 iops : min= 15, max= 44, avg=32.92, stdev= 8.99, samples=95 lat (msec) : 2=1.07%, 4=36.91%, 10=6.38%, 20=18.13%, 50=35.90% lat (msec) : 100=1.56%, 250=0.03% cpu : usr=0.09%, sys=4.72%, ctx=3331, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30259: Tue Apr 16 15:46:44 2024 read: IOPS=31, BW=127KiB/s (130kB/s)(6028KiB/47387msec) clat (usec): min=1749, max=48625, avg=4283.53, stdev=4099.38 lat (usec): min=1750, max=48625, avg=4284.49, stdev=4099.39 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2802], 60.00th=[ 3032], | 70.00th=[ 3392], 80.00th=[ 4752], 90.00th=[ 9241], 95.00th=[12256], | 99.00th=[21103], 99.50th=[27132], 99.90th=[42206], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 23, max= 240, per=25.39%, avg=127.20, stdev=49.64, samples=94 iops : min= 5, max= 60, avg=31.78, stdev=12.43, samples=94 write: IOPS=33, BW=132KiB/s (135kB/s)(6260KiB/47387msec) clat (msec): min=12, max=186, avg=26.13, stdev=10.38 lat (msec): min=12, max=186, avg=26.13, stdev=10.38 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 25], 60.00th=[ 27], | 70.00th=[ 29], 80.00th=[ 31], 90.00th=[ 37], 95.00th=[ 46], | 99.00th=[ 62], 99.50th=[ 71], 99.90th=[ 112], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 63, max= 184, per=25.17%, avg=131.88, stdev=36.95, samples=94 iops : min= 15, max= 46, avg=32.95, stdev= 9.26, samples=94 lat (msec) : 2=1.37%, 4=36.04%, 10=7.78%, 20=17.68%, 50=35.48% lat (msec) : 100=1.60%, 250=0.07% cpu : usr=0.11%, sys=4.75%, ctx=3361, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30260: Tue Apr 16 15:46:44 2024 read: IOPS=31, BW=128KiB/s (131kB/s)(6052KiB/47323msec) clat (usec): min=1738, max=42316, avg=4062.15, stdev=3515.88 lat (usec): min=1739, max=42317, avg=4063.12, stdev=3515.89 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2769], 60.00th=[ 2999], | 70.00th=[ 3294], 80.00th=[ 4359], 90.00th=[ 8291], 95.00th=[11600], | 99.00th=[17171], 99.50th=[18482], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 8, max= 271, per=25.49%, avg=127.69, stdev=55.40, samples=94 iops : min= 2, max= 67, avg=31.82, stdev=13.85, samples=94 write: IOPS=32, BW=132KiB/s (135kB/s)(6236KiB/47323msec) clat (msec): min=12, max=186, avg=26.39, stdev=10.44 lat (msec): min=12, max=186, avg=26.39, stdev=10.44 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 25], 60.00th=[ 27], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 39], 95.00th=[ 46], | 99.00th=[ 61], 99.50th=[ 72], 99.90th=[ 84], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 56, max= 184, per=25.12%, avg=131.61, stdev=36.23, samples=94 iops : min= 14, max= 46, avg=32.80, stdev= 9.06, samples=94 lat (msec) : 2=1.37%, 4=37.14%, 10=7.10%, 20=17.35%, 50=35.48% lat (msec) : 100=1.53%, 250=0.03% cpu : usr=0.09%, sys=4.79%, ctx=3323, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=501KiB/s (513kB/s), 124KiB/s-128KiB/s (127kB/s-131kB/s), io=23.4MiB (24.6MB), run=47323-47897msec WRITE: bw=525KiB/s (537kB/s), 132KiB/s-133KiB/s (135kB/s-136kB/s), io=24.6MiB (25.7MB), run=47323-47897msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30312: Tue Apr 16 15:46:51 2024 read: IOPS=64, BW=1033KiB/s (1057kB/s)(5920KiB/5733msec) clat (usec): min=51, max=38618, avg=11243.45, stdev=7002.61 lat (usec): min=52, max=38619, avg=11244.22, stdev=7002.61 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 78], 10.00th=[ 102], 20.00th=[ 153], | 30.00th=[10290], 40.00th=[11469], 50.00th=[12518], 60.00th=[13566], | 70.00th=[14615], 80.00th=[16057], 90.00th=[17957], 95.00th=[19792], | 99.00th=[29492], 99.50th=[34866], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 704, max= 1184, per=23.77%, avg=989.09, stdev=157.71, samples=11 iops : min= 44, max= 74, avg=61.82, stdev= 9.86, samples=11 write: IOPS=69, BW=1111KiB/s (1137kB/s)(6368KiB/5733msec) clat (usec): min=670, max=35578, avg=3927.24, stdev=4622.69 lat (usec): min=671, max=35579, avg=3928.59, stdev=4622.73 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 832], 10.00th=[ 914], 20.00th=[ 1074], | 30.00th=[ 1254], 40.00th=[ 1434], 50.00th=[ 2114], 60.00th=[ 3458], | 70.00th=[ 4293], 80.00th=[ 5669], 90.00th=[ 8979], 95.00th=[11469], | 99.00th=[27395], 99.50th=[31327], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 736, max= 1536, per=24.45%, avg=1070.55, stdev=288.84, samples=11 iops : min= 46, max= 96, avg=66.91, stdev=18.05, samples=11 lat (usec) : 100=4.56%, 250=5.60%, 750=0.78%, 1000=7.42% lat (msec) : 2=17.58%, 4=9.24%, 10=17.32%, 20=34.24%, 50=3.26% cpu : usr=0.12%, sys=23.50%, ctx=2147, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30313: Tue Apr 16 15:46:51 2024 read: IOPS=63, BW=1014KiB/s (1039kB/s)(5744KiB/5663msec) clat (usec): min=56, max=35928, avg=11035.57, stdev=6583.33 lat (usec): min=57, max=35930, avg=11036.35, stdev=6583.35 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 79], 10.00th=[ 98], 20.00th=[ 135], | 30.00th=[10421], 40.00th=[11600], 50.00th=[12649], 60.00th=[13566], | 70.00th=[14746], 80.00th=[15926], 90.00th=[17957], 95.00th=[19268], | 99.00th=[23987], 99.50th=[26346], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 576, max= 1312, per=24.12%, avg=1003.45, stdev=202.16, samples=11 iops : min= 36, max= 82, avg=62.64, stdev=12.70, samples=11 write: IOPS=72, BW=1156KiB/s (1183kB/s)(6544KiB/5663msec) clat (usec): min=669, max=58655, avg=4135.36, stdev=5393.72 lat (usec): min=670, max=58657, avg=4136.77, stdev=5393.66 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 807], 10.00th=[ 865], 20.00th=[ 1090], | 30.00th=[ 1270], 40.00th=[ 1385], 50.00th=[ 2409], 60.00th=[ 3458], | 70.00th=[ 4293], 80.00th=[ 5800], 90.00th=[ 9372], 95.00th=[12256], | 99.00th=[25297], 99.50th=[32375], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 704, max= 1696, per=25.97%, avg=1137.18, stdev=283.91, samples=11 iops : min= 44, max= 106, avg=71.00, stdev=17.67, samples=11 lat (usec) : 100=5.08%, 250=5.21%, 750=1.04%, 1000=7.42% lat (msec) : 2=17.58%, 4=9.64%, 10=15.10%, 20=36.20%, 50=2.60% lat (msec) : 100=0.13% cpu : usr=0.25%, sys=22.77%, ctx=2138, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30314: Tue Apr 16 15:46:51 2024 read: IOPS=68, BW=1091KiB/s (1117kB/s)(6256KiB/5734msec) clat (usec): min=51, max=31014, avg=10951.16, stdev=6812.30 lat (usec): min=51, max=31014, avg=10951.93, stdev=6812.31 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 87], 10.00th=[ 111], 20.00th=[ 149], | 30.00th=[ 9896], 40.00th=[11207], 50.00th=[12256], 60.00th=[13173], | 70.00th=[14484], 80.00th=[15926], 90.00th=[18220], 95.00th=[20579], | 99.00th=[30016], 99.50th=[30278], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 768, max= 1664, per=26.21%, avg=1090.73, stdev=259.09, samples=11 iops : min= 48, max= 104, avg=68.09, stdev=16.25, samples=11 write: IOPS=65, BW=1052KiB/s (1077kB/s)(6032KiB/5734msec) clat (usec): min=715, max=25764, avg=3819.18, stdev=4071.21 lat (usec): min=716, max=25765, avg=3820.98, stdev=4071.68 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 824], 10.00th=[ 906], 20.00th=[ 1106], | 30.00th=[ 1254], 40.00th=[ 1352], 50.00th=[ 1614], 60.00th=[ 3195], | 70.00th=[ 4359], 80.00th=[ 6259], 90.00th=[ 8586], 95.00th=[11863], | 99.00th=[20579], 99.50th=[21890], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 544, max= 1632, per=23.25%, avg=1017.91, stdev=305.31, samples=11 iops : min= 34, max= 102, avg=63.55, stdev=19.02, samples=11 lat (usec) : 100=4.30%, 250=6.77%, 750=0.52%, 1000=6.64% lat (msec) : 2=18.62%, 4=7.68%, 10=16.54%, 20=35.55%, 50=3.39% cpu : usr=0.16%, sys=22.68%, ctx=2206, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30315: Tue Apr 16 15:46:51 2024 read: IOPS=65, BW=1048KiB/s (1073kB/s)(6032KiB/5756msec) clat (usec): min=52, max=55214, avg=11224.87, stdev=7483.69 lat (usec): min=52, max=55215, avg=11225.65, stdev=7483.70 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 74], 10.00th=[ 101], 20.00th=[ 133], | 30.00th=[ 9634], 40.00th=[11338], 50.00th=[12387], 60.00th=[13435], | 70.00th=[14877], 80.00th=[16319], 90.00th=[18744], 95.00th=[22676], | 99.00th=[27919], 99.50th=[32900], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 832, max= 1472, per=24.82%, avg=1032.73, stdev=177.07, samples=11 iops : min= 52, max= 92, avg=64.55, stdev=11.07, samples=11 write: IOPS=67, BW=1087KiB/s (1113kB/s)(6256KiB/5756msec) clat (usec): min=721, max=46020, avg=3874.07, stdev=4158.51 lat (usec): min=722, max=46021, avg=3875.38, stdev=4158.56 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 840], 10.00th=[ 938], 20.00th=[ 1090], | 30.00th=[ 1237], 40.00th=[ 1385], 50.00th=[ 1926], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 6521], 90.00th=[ 8848], 95.00th=[10290], | 99.00th=[17695], 99.50th=[24249], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 928, max= 1376, per=24.52%, avg=1073.45, stdev=159.59, samples=11 iops : min= 58, max= 86, avg=67.09, stdev= 9.97, samples=11 lat (usec) : 100=4.82%, 250=6.25%, 500=0.26%, 750=0.52%, 1000=7.68% lat (msec) : 2=17.45%, 4=7.03%, 10=19.27%, 20=32.29%, 50=4.30% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=22.36%, ctx=2201, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4161KiB/s (4261kB/s), 1014KiB/s-1091KiB/s (1039kB/s-1117kB/s), io=23.4MiB (24.5MB), run=5663-5756msec WRITE: bw=4378KiB/s (4483kB/s), 1052KiB/s-1156KiB/s (1077kB/s-1183kB/s), io=24.6MiB (25.8MB), run=5663-5756msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30308: Tue Apr 16 15:46:58 2024 read: IOPS=28, BW=460KiB/s (471kB/s)(5920KiB/12883msec) clat (usec): min=2053, max=31279, avg=4930.43, stdev=4338.79 lat (usec): min=2053, max=31280, avg=4931.29, stdev=4338.79 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2311], 10.00th=[ 2442], 20.00th=[ 2573], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3261], 60.00th=[ 3589], | 70.00th=[ 4146], 80.00th=[ 5669], 90.00th=[10028], 95.00th=[15008], | 99.00th=[23987], 99.50th=[25035], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 288, max= 896, per=24.35%, avg=446.52, stdev=140.71, samples=25 iops : min= 18, max= 56, avg=27.80, stdev= 8.75, samples=25 write: IOPS=30, BW=494KiB/s (506kB/s)(6368KiB/12883msec) clat (usec): min=14101, max=77720, avg=27763.94, stdev=8694.50 lat (usec): min=14102, max=77722, avg=27765.75, stdev=8694.64 clat percentiles (usec): | 1.00th=[15139], 5.00th=[16909], 10.00th=[17695], 20.00th=[20055], | 30.00th=[22414], 40.00th=[24773], 50.00th=[26608], 60.00th=[28181], | 70.00th=[30802], 80.00th=[33817], 90.00th=[39584], 95.00th=[44827], | 99.00th=[51643], 99.50th=[67634], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 320, max= 640, per=25.32%, avg=488.76, stdev=106.97, samples=25 iops : min= 20, max= 40, avg=30.44, stdev= 6.64, samples=25 lat (msec) : 4=33.07%, 10=10.03%, 20=14.58%, 50=41.67%, 100=0.65% cpu : usr=0.12%, sys=5.20%, ctx=857, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30309: Tue Apr 16 15:46:58 2024 read: IOPS=27, BW=440KiB/s (451kB/s)(5744KiB/13055msec) clat (usec): min=2014, max=42510, avg=4991.03, stdev=5108.34 lat (usec): min=2015, max=42511, avg=4991.92, stdev=5108.34 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2376], 10.00th=[ 2507], 20.00th=[ 2638], | 30.00th=[ 2835], 40.00th=[ 3032], 50.00th=[ 3228], 60.00th=[ 3458], | 70.00th=[ 4015], 80.00th=[ 4948], 90.00th=[ 9503], 95.00th=[16057], | 99.00th=[27919], 99.50th=[38011], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 160, max= 832, per=23.94%, avg=439.04, stdev=174.67, samples=25 iops : min= 10, max= 52, avg=27.44, stdev=10.92, samples=25 write: IOPS=31, BW=501KiB/s (513kB/s)(6544KiB/13055msec) clat (usec): min=12588, max=68341, avg=27516.01, stdev=8693.31 lat (usec): min=12588, max=68343, avg=27517.76, stdev=8693.42 clat percentiles (usec): | 1.00th=[15008], 5.00th=[16319], 10.00th=[17695], 20.00th=[19792], | 30.00th=[22414], 40.00th=[24773], 50.00th=[26346], 60.00th=[28181], | 70.00th=[30278], 80.00th=[33162], 90.00th=[38536], 95.00th=[44303], | 99.00th=[57410], 99.50th=[59507], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 320, max= 672, per=25.40%, avg=490.24, stdev=105.62, samples=25 iops : min= 20, max= 42, avg=30.64, stdev= 6.60, samples=25 lat (msec) : 4=32.68%, 10=9.90%, 20=14.06%, 50=42.32%, 100=1.04% cpu : usr=0.08%, sys=5.17%, ctx=863, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30310: Tue Apr 16 15:46:58 2024 read: IOPS=30, BW=496KiB/s (508kB/s)(6256KiB/12617msec) clat (usec): min=2100, max=37294, avg=5113.56, stdev=4809.48 lat (usec): min=2100, max=37294, avg=5114.47, stdev=4809.56 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2376], 10.00th=[ 2474], 20.00th=[ 2671], | 30.00th=[ 2802], 40.00th=[ 2966], 50.00th=[ 3261], 60.00th=[ 3556], | 70.00th=[ 4146], 80.00th=[ 6325], 90.00th=[ 9896], 95.00th=[14615], | 99.00th=[26608], 99.50th=[31851], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 192, max= 960, per=27.20%, avg=498.92, stdev=209.65, samples=25 iops : min= 12, max= 60, avg=31.00, stdev=13.15, samples=25 write: IOPS=29, BW=478KiB/s (490kB/s)(6032KiB/12617msec) clat (usec): min=14289, max=62894, avg=28138.90, stdev=8258.80 lat (usec): min=14290, max=62896, avg=28140.57, stdev=8258.88 clat percentiles (usec): | 1.00th=[15270], 5.00th=[17171], 10.00th=[18220], 20.00th=[20579], | 30.00th=[23462], 40.00th=[25560], 50.00th=[27132], 60.00th=[28967], | 70.00th=[30802], 80.00th=[34341], 90.00th=[39584], 95.00th=[43779], | 99.00th=[52691], 99.50th=[58983], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 320, max= 638, per=24.65%, avg=475.84, stdev=103.20, samples=25 iops : min= 20, max= 39, avg=29.56, stdev= 6.40, samples=25 lat (msec) : 4=34.24%, 10=11.59%, 20=12.11%, 50=41.28%, 100=0.78% cpu : usr=0.10%, sys=5.35%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30311: Tue Apr 16 15:46:58 2024 read: IOPS=29, BW=471KiB/s (483kB/s)(6032KiB/12798msec) clat (usec): min=2063, max=26215, avg=5035.64, stdev=3923.41 lat (usec): min=2064, max=26216, avg=5036.73, stdev=3923.44 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2376], 10.00th=[ 2507], 20.00th=[ 2737], | 30.00th=[ 2933], 40.00th=[ 3130], 50.00th=[ 3392], 60.00th=[ 3916], | 70.00th=[ 4621], 80.00th=[ 6456], 90.00th=[ 9896], 95.00th=[13960], | 99.00th=[23462], 99.50th=[25822], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 224, max= 736, per=25.18%, avg=461.80, stdev=145.34, samples=25 iops : min= 14, max= 46, avg=28.68, stdev= 9.04, samples=25 write: IOPS=30, BW=489KiB/s (501kB/s)(6256KiB/12798msec) clat (usec): min=14690, max=63226, avg=27852.58, stdev=8000.71 lat (usec): min=14692, max=63228, avg=27854.25, stdev=8000.78 clat percentiles (usec): | 1.00th=[15401], 5.00th=[16712], 10.00th=[17957], 20.00th=[20579], | 30.00th=[23200], 40.00th=[25560], 50.00th=[27132], 60.00th=[28705], | 70.00th=[31327], 80.00th=[33817], 90.00th=[38536], 95.00th=[42206], | 99.00th=[49546], 99.50th=[60556], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 288, max= 608, per=25.25%, avg=487.36, stdev=101.58, samples=25 iops : min= 18, max= 38, avg=30.28, stdev= 6.32, samples=25 lat (msec) : 4=30.47%, 10=13.80%, 20=13.80%, 50=41.54%, 100=0.39% cpu : usr=0.05%, sys=5.20%, ctx=866, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1835KiB/s (1879kB/s), 440KiB/s-496KiB/s (451kB/s-508kB/s), io=23.4MiB (24.5MB), run=12617-13055msec WRITE: bw=1930KiB/s (1977kB/s), 478KiB/s-501KiB/s (490kB/s-513kB/s), io=24.6MiB (25.8MB), run=12617-13055msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30332: Tue Apr 16 15:47:01 2024 read: IOPS=7, BW=7405KiB/s (7583kB/s)(7168KiB/968msec) clat (msec): min=17, max=335, avg=92.89, stdev=110.91 lat (msec): min=17, max=335, avg=92.89, stdev=110.91 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 33], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 52], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 109], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 6131, max= 6131, per=73.77%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5289KiB/s (5416kB/s)(5120KiB/968msec) clat (usec): min=37890, max=79384, avg=63309.03, stdev=16573.51 lat (usec): min=37961, max=79457, avg=63380.59, stdev=16574.48 clat percentiles (usec): | 1.00th=[38011], 5.00th=[38011], 10.00th=[38011], 20.00th=[38011], | 30.00th=[55837], 40.00th=[55837], 50.00th=[69731], 60.00th=[69731], | 70.00th=[72877], 80.00th=[72877], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] lat (msec) : 20=8.33%, 50=25.00%, 100=50.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=11.27%, ctx=35, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30333: Tue Apr 16 15:47:01 2024 read: IOPS=1, BW=2028KiB/s (2076kB/s)(2048KiB/1010msec) clat (msec): min=23, max=411, avg=217.30, stdev=274.56 lat (msec): min=23, max=411, avg=217.30, stdev=274.56 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 2035, max= 2048, per=24.56%, avg=2041.50, stdev= 9.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=9, BW=9.90MiB/s (10.4MB/s)(10.0MiB/1010msec) clat (msec): min=40, max=107, avg=57.28, stdev=22.31 lat (msec): min=41, max=107, avg=57.35, stdev=22.33 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 48], 50.00th=[ 49], 60.00th=[ 50], | 70.00th=[ 56], 80.00th=[ 56], 90.00th=[ 89], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 8192, max=10178, per=78.94%, avg=9185.00, stdev=1404.31, samples=2 iops : min= 8, max= 9, avg= 8.50, stdev= 0.71, samples=2 lat (msec) : 50=66.67%, 100=16.67%, 250=8.33%, 500=8.33% cpu : usr=0.10%, sys=13.08%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30334: Tue Apr 16 15:47:01 2024 read: IOPS=2, BW=2494KiB/s (2553kB/s)(6144KiB/2464msec) clat (msec): min=45, max=405, avg=118.60, stdev=141.65 lat (msec): min=45, max=405, avg=118.61, stdev=141.65 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 71], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 3465, max= 6131, per=57.73%, avg=4798.00, stdev=1885.15, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2494KiB/s (2553kB/s)(6144KiB/2464msec) clat (msec): min=65, max=848, avg=291.82, stdev=310.51 lat (msec): min=65, max=849, avg=291.87, stdev=310.53 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 148], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 1732, max= 4087, per=28.38%, avg=3302.00, stdev=1359.66, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 lat (msec) : 50=25.00%, 100=33.33%, 250=16.67%, 500=16.67%, 1000=8.33% cpu : usr=0.00%, sys=5.97%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30335: Tue Apr 16 15:47:01 2024 read: IOPS=4, BW=5074KiB/s (5196kB/s)(5120KiB/1009msec) clat (usec): min=18064, max=52144, avg=27557.06, stdev=13963.03 lat (usec): min=18065, max=52146, avg=27558.12, stdev=13963.19 clat percentiles (usec): | 1.00th=[17957], 5.00th=[17957], 10.00th=[17957], 20.00th=[17957], | 30.00th=[20317], 40.00th=[20317], 50.00th=[22938], 60.00th=[22938], | 70.00th=[24511], 80.00th=[24511], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 8192, max= 8192, per=98.57%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7104KiB/s (7275kB/s)(7168KiB/1009msec) clat (msec): min=32, max=435, avg=123.98, stdev=140.50 lat (msec): min=32, max=435, avg=124.06, stdev=140.50 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 50], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 73], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 126], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 4071, max=10240, per=61.49%, avg=7155.50, stdev=4362.14, samples=2 iops : min= 3, max= 10, avg= 6.50, stdev= 4.95, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=12.00%, ctx=32, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8312KiB/s (8511kB/s), 2028KiB/s-7405KiB/s (2076kB/s-7583kB/s), io=20.0MiB (20.0MB), run=968-2464msec WRITE: bw=11.4MiB/s (11.9MB/s), 2494KiB/s-9.90MiB/s (2553kB/s-10.4MB/s), io=28.0MiB (29.4MB), run=968-2464msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30336: Tue Apr 16 15:47:02 2024 read: IOPS=11, BW=11.7MiB/s (12.3MB/s)(7168KiB/598msec) clat (usec): min=440, max=335828, avg=65209.03, stdev=121269.72 lat (usec): min=440, max=335828, avg=65210.30, stdev=121269.57 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], 20.00th=[ 515], | 30.00th=[ 4178], 40.00th=[ 4178], 50.00th=[ 18744], 60.00th=[ 39060], | 70.00th=[ 39060], 80.00th=[ 57410], 90.00th=[337642], 95.00th=[337642], | 99.00th=[337642], 99.50th=[337642], 99.90th=[337642], 99.95th=[337642], | 99.99th=[337642] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8562KiB/s (8767kB/s)(5120KiB/598msec) clat (usec): min=19817, max=43442, avg=25771.93, stdev=9928.81 lat (usec): min=19834, max=43460, avg=25811.86, stdev=9917.55 clat percentiles (usec): | 1.00th=[19792], 5.00th=[19792], 10.00th=[19792], 20.00th=[19792], | 30.00th=[21365], 40.00th=[21365], 50.00th=[21890], 60.00th=[21890], | 70.00th=[22676], 80.00th=[22676], 90.00th=[43254], 95.00th=[43254], | 99.00th=[43254], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 4096, max= 4096, per=48.05%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 10=8.33%, 20=16.67%, 50=41.67%, 100=8.33%, 500=8.33% cpu : usr=0.00%, sys=44.39%, ctx=182, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30337: Tue Apr 16 15:47:02 2024 read: IOPS=3, BW=3495KiB/s (3579kB/s)(2048KiB/586msec) clat (usec): min=58811, max=79412, avg=69112.12, stdev=14566.97 lat (usec): min=58812, max=79413, avg=69113.20, stdev=14567.11 clat percentiles (usec): | 1.00th=[58983], 5.00th=[58983], 10.00th=[58983], 20.00th=[58983], | 30.00th=[58983], 40.00th=[58983], 50.00th=[58983], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 4079, max= 4079, per=66.99%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=17, BW=17.1MiB/s (17.9MB/s)(10.0MiB/586msec) clat (usec): min=24245, max=81761, avg=44612.97, stdev=17472.99 lat (usec): min=24324, max=81789, avg=44690.57, stdev=17453.23 clat percentiles (usec): | 1.00th=[24249], 5.00th=[24249], 10.00th=[24249], 20.00th=[28181], | 30.00th=[28181], 40.00th=[39584], 50.00th=[42730], 60.00th=[42730], | 70.00th=[50594], 80.00th=[50594], 90.00th=[63177], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min=14278, max=14278, per=100.00%, avg=14278.00, stdev= 0.00, samples=1 iops : min= 13, max= 13, avg=13.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=41.67% cpu : usr=0.17%, sys=55.38%, ctx=72, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30338: Tue Apr 16 15:47:02 2024 read: IOPS=2, BW=2132KiB/s (2183kB/s)(6144KiB/2882msec) clat (usec): min=526, max=500359, avg=245973.50, stdev=237157.79 lat (usec): min=527, max=500360, avg=245974.74, stdev=237158.00 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 46400], | 30.00th=[ 46400], 40.00th=[ 50594], 50.00th=[ 50594], 60.00th=[396362], | 70.00th=[484443], 80.00th=[484443], 90.00th=[501220], 95.00th=[501220], | 99.00th=[501220], 99.50th=[501220], 99.90th=[501220], 99.95th=[501220], | 99.99th=[501220] bw ( KiB/s): min= 2048, max= 4096, per=56.06%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2132KiB/s (2183kB/s)(6144KiB/2882msec) clat (msec): min=61, max=542, avg=232.41, stdev=218.61 lat (msec): min=61, max=542, avg=232.47, stdev=218.61 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 92], 50.00th=[ 92], 60.00th=[ 129], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 2048, max= 4096, per=36.04%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 100=33.33%, 250=8.33%, 500=25.00%, 750=16.67% cpu : usr=0.00%, sys=26.59%, ctx=557, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30339: Tue Apr 16 15:47:02 2024 read: IOPS=1, BW=1522KiB/s (1559kB/s)(5120KiB/3363msec) clat (msec): min=51, max=741, avg=387.70, stdev=289.23 lat (msec): min=51, max=741, avg=387.70, stdev=289.23 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 2048, max= 4096, per=44.85%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=2, BW=2131KiB/s (2183kB/s)(7168KiB/3363msec) clat (msec): min=47, max=458, avg=202.65, stdev=180.05 lat (msec): min=47, max=458, avg=202.72, stdev=180.06 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 62], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 95], 60.00th=[ 268], | 70.00th=[ 268], 80.00th=[ 426], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 2048, max= 4096, per=42.04%, avg=3584.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 3.50, stdev= 1.00, samples=4 lat (msec) : 50=8.33%, 100=33.33%, 250=8.33%, 500=33.33%, 750=16.67% cpu : usr=0.09%, sys=32.90%, ctx=680, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6090KiB/s (6236kB/s), 1522KiB/s-11.7MiB/s (1559kB/s-12.3MB/s), io=20.0MiB (20.0MB), run=586-3363msec WRITE: bw=8526KiB/s (8730kB/s), 2131KiB/s-17.1MiB/s (2183kB/s-17.9MB/s), io=28.0MiB (29.4MB), run=586-3363msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30349: Tue Apr 16 15:47:06 2024 read: IOPS=1, BW=5626KiB/s (5761kB/s)(12.0MiB/2184msec) clat (msec): min=177, max=1245, avg=727.58, stdev=534.86 lat (msec): min=177, max=1245, avg=727.58, stdev=534.86 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 760], 50.00th=[ 760], 60.00th=[ 760], | 70.00th=[ 1250], 80.00th=[ 1250], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.05%, sys=45.85%, ctx=850, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30351: Tue Apr 16 15:47:06 2024 read: IOPS=0, BW=1621KiB/s (1660kB/s)(4096KiB/2527msec) clat (nsec): min=177928k, max=177928k, avg=177927674.00, stdev= 0.00 lat (nsec): min=177929k, max=177929k, avg=177929081.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3242KiB/s (3320kB/s)(8192KiB/2527msec) clat (msec): min=643, max=1690, avg=1166.79, stdev=739.97 lat (msec): min=643, max=1690, avg=1167.07, stdev=739.98 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 1687], | 70.00th=[ 1687], 80.00th=[ 1687], 90.00th=[ 1687], 95.00th=[ 1687], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 8192, max= 8192, per=63.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=10.72%, ctx=121, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30353: Tue Apr 16 15:47:06 2024 write: IOPS=1, BW=4851KiB/s (4968kB/s)(12.0MiB/2533msec) clat (msec): min=81, max=1629, avg=839.35, stdev=774.32 lat (msec): min=82, max=1629, avg=839.66, stdev=774.31 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 8192, max= 8192, per=63.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 1000=33.33% cpu : usr=0.04%, sys=10.43%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30355: Tue Apr 16 15:47:06 2024 write: IOPS=1, BW=4977KiB/s (5096kB/s)(12.0MiB/2469msec) clat (msec): min=725, max=1000, avg=819.72, stdev=156.28 lat (msec): min=725, max=1000, avg=819.98, stdev=156.27 clat percentiles (msec): | 1.00th=[ 726], 5.00th=[ 726], 10.00th=[ 726], 20.00th=[ 726], | 30.00th=[ 726], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 735], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 8192, max= 8192, per=63.33%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67% cpu : usr=0.00%, sys=11.02%, ctx=81, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6484KiB/s (6639kB/s), 1621KiB/s-5626KiB/s (1660kB/s-5761kB/s), io=16.0MiB (16.8MB), run=2184-2527msec WRITE: bw=12.6MiB/s (13.2MB/s), 3242KiB/s-4977KiB/s (3320kB/s-5096kB/s), io=32.0MiB (33.6MB), run=2469-2533msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30348: Tue Apr 16 15:47:06 2024 read: IOPS=1, BW=4211KiB/s (4312kB/s)(12.0MiB/2918msec) clat (msec): min=67, max=1957, avg=968.62, stdev=947.93 lat (msec): min=67, max=1957, avg=968.62, stdev=947.93 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 877], 50.00th=[ 877], 60.00th=[ 877], | 70.00th=[ 1955], 80.00th=[ 1955], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 1000=33.33% cpu : usr=0.00%, sys=3.60%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30350: Tue Apr 16 15:47:06 2024 read: IOPS=0, BW=1367KiB/s (1400kB/s)(4096KiB/2996msec) clat (nsec): min=182116k, max=182116k, avg=182116493.00, stdev= 0.00 lat (nsec): min=182119k, max=182119k, avg=182118514.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2734KiB/s (2800kB/s)(8192KiB/2996msec) clat (msec): min=823, max=1984, avg=1404.01, stdev=820.70 lat (msec): min=823, max=1984, avg=1404.28, stdev=820.69 clat percentiles (msec): | 1.00th=[ 827], 5.00th=[ 827], 10.00th=[ 827], 20.00th=[ 827], | 30.00th=[ 827], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 1989], | 70.00th=[ 1989], 80.00th=[ 1989], 90.00th=[ 1989], 95.00th=[ 1989], | 99.00th=[ 1989], 99.50th=[ 1989], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 5603, max= 5603, per=51.23%, avg=5603.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=4.27%, ctx=61, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30352: Tue Apr 16 15:47:06 2024 write: IOPS=1, BW=4114KiB/s (4213kB/s)(12.0MiB/2987msec) clat (msec): min=71, max=1972, avg=994.07, stdev=951.69 lat (msec): min=71, max=1972, avg=994.44, stdev=951.71 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 936], 50.00th=[ 936], 60.00th=[ 936], | 70.00th=[ 1972], 80.00th=[ 1972], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 8192, max= 8192, per=74.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 1000=33.33% cpu : usr=0.03%, sys=3.48%, ctx=17, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30354: Tue Apr 16 15:47:06 2024 write: IOPS=2, BW=11.2MiB/s (11.7MB/s)(12.0MiB/1076msec) clat (msec): min=70, max=827, avg=339.84, stdev=423.39 lat (msec): min=70, max=828, avg=340.21, stdev=423.34 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.09%, sys=11.26%, ctx=41, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5469KiB/s (5600kB/s), 1367KiB/s-4211KiB/s (1400kB/s-4312kB/s), io=16.0MiB (16.8MB), run=2918-2996msec WRITE: bw=10.7MiB/s (11.2MB/s), 2734KiB/s-11.2MiB/s (2800kB/s-11.7MB/s), io=32.0MiB (33.6MB), run=1076-2996msec