== sanity test 398b: DIO and buffer IO race ============== 17:13:25 (1713302005) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.38172 s, 36.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31576: Tue Apr 16 17:13:40 2024 read: IOPS=121, BW=486KiB/s (498kB/s)(5992KiB/12328msec) clat (usec): min=44, max=177356, avg=3696.28, stdev=5483.16 lat (usec): min=45, max=177357, avg=3697.36, stdev=5483.04 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 182], | 30.00th=[ 2212], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7373], 95.00th=[ 8717], | 99.00th=[ 12518], 99.50th=[ 15139], 99.90th=[ 45876], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 288, max= 704, per=24.86%, avg=479.29, stdev=99.34, samples=24 iops : min= 72, max= 176, avg=119.79, stdev=24.86, samples=24 write: IOPS=127, BW=511KiB/s (523kB/s)(6296KiB/12328msec) clat (usec): min=437, max=42473, avg=4243.17, stdev=4293.64 lat (usec): min=437, max=42474, avg=4243.90, stdev=4293.71 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 523], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 938], 40.00th=[ 2212], 50.00th=[ 3392], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6849], 90.00th=[ 9634], 95.00th=[12125], | 99.00th=[17433], 99.50th=[21627], 99.90th=[41681], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 352, max= 712, per=24.81%, avg=500.58, stdev=89.32, samples=24 iops : min= 88, max= 178, avg=125.12, stdev=22.32, samples=24 lat (usec) : 50=0.36%, 100=8.69%, 250=0.81%, 500=0.85%, 750=12.70% lat (usec) : 1000=2.60% lat (msec) : 2=5.92%, 4=29.33%, 10=32.78%, 20=5.47%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=23.81%, ctx=5774, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31578: Tue Apr 16 17:13:40 2024 read: IOPS=119, BW=477KiB/s (488kB/s)(5940KiB/12454msec) clat (usec): min=46, max=37421, avg=3415.00, stdev=2982.88 lat (usec): min=46, max=37421, avg=3415.73, stdev=2982.84 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 65], 10.00th=[ 72], 20.00th=[ 90], | 30.00th=[ 2180], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3425], | 70.00th=[ 4228], 80.00th=[ 5669], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[11338], 99.50th=[14615], 99.90th=[31065], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 296, max= 648, per=24.33%, avg=469.04, stdev=84.78, samples=24 iops : min= 74, max= 162, avg=117.12, stdev=21.23, samples=24 write: IOPS=127, BW=510KiB/s (522kB/s)(6348KiB/12454msec) clat (usec): min=424, max=171047, avg=4581.49, stdev=5940.27 lat (usec): min=425, max=171048, avg=4582.16, stdev=5940.27 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 635], | 30.00th=[ 1106], 40.00th=[ 2769], 50.00th=[ 3785], 60.00th=[ 4686], | 70.00th=[ 5800], 80.00th=[ 7308], 90.00th=[ 9765], 95.00th=[ 12256], | 99.00th=[ 17695], 99.50th=[ 26608], 99.90th=[ 42730], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 352, max= 608, per=24.56%, avg=495.67, stdev=72.97, samples=24 iops : min= 88, max= 152, avg=123.79, stdev=18.25, samples=24 lat (usec) : 50=0.78%, 100=9.44%, 250=0.55%, 500=0.88%, 750=11.88% lat (usec) : 1000=2.28% lat (msec) : 2=5.14%, 4=29.30%, 10=33.98%, 20=5.24%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.20%, ctx=5790, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31580: Tue Apr 16 17:13:40 2024 read: IOPS=121, BW=487KiB/s (499kB/s)(6028KiB/12380msec) clat (usec): min=46, max=40763, avg=3572.06, stdev=3140.71 lat (usec): min=46, max=40765, avg=3572.71, stdev=3140.69 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 109], | 30.00th=[ 2245], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5604], 90.00th=[ 7308], 95.00th=[ 8979], | 99.00th=[11600], 99.50th=[14091], 99.90th=[35914], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 200, max= 680, per=25.00%, avg=482.00, stdev=95.95, samples=24 iops : min= 50, max= 170, avg=120.50, stdev=23.99, samples=24 write: IOPS=126, BW=506KiB/s (518kB/s)(6260KiB/12380msec) clat (usec): min=462, max=202094, avg=4398.00, stdev=6485.84 lat (usec): min=463, max=202096, avg=4399.19, stdev=6485.63 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 963], 40.00th=[ 2343], 50.00th=[ 3523], 60.00th=[ 4359], | 70.00th=[ 5342], 80.00th=[ 7111], 90.00th=[ 9634], 95.00th=[ 11863], | 99.00th=[ 17433], 99.50th=[ 23725], 99.90th=[ 37487], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 280, max= 672, per=24.64%, avg=497.33, stdev=95.00, samples=24 iops : min= 70, max= 168, avg=124.33, stdev=23.75, samples=24 lat (usec) : 50=0.46%, 100=9.18%, 250=0.91%, 500=1.01%, 750=12.43% lat (usec) : 1000=2.41% lat (msec) : 2=5.31%, 4=28.32%, 10=33.59%, 20=5.96%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=23.52%, ctx=5775, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31582: Tue Apr 16 17:13:40 2024 read: IOPS=123, BW=493KiB/s (505kB/s)(6052KiB/12280msec) clat (usec): min=46, max=30747, avg=3473.78, stdev=2790.40 lat (usec): min=47, max=30747, avg=3475.58, stdev=2791.81 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 147], | 30.00th=[ 2212], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 7111], 95.00th=[ 8356], | 99.00th=[10945], 99.50th=[13698], 99.90th=[22414], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 264, max= 592, per=25.31%, avg=488.00, stdev=78.70, samples=24 iops : min= 66, max= 148, avg=122.00, stdev=19.68, samples=24 write: IOPS=126, BW=508KiB/s (520kB/s)(6236KiB/12280msec) clat (usec): min=432, max=170863, avg=4432.77, stdev=5939.44 lat (usec): min=433, max=170864, avg=4433.81, stdev=5939.55 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 971], 40.00th=[ 2147], 50.00th=[ 3490], 60.00th=[ 4359], | 70.00th=[ 5538], 80.00th=[ 7111], 90.00th=[ 10028], 95.00th=[ 12387], | 99.00th=[ 18744], 99.50th=[ 22676], 99.90th=[ 35914], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 416, max= 600, per=24.94%, avg=503.33, stdev=55.17, samples=24 iops : min= 104, max= 150, avg=125.83, stdev=13.79, samples=24 lat (usec) : 50=0.52%, 100=8.59%, 250=1.01%, 500=0.81%, 750=12.37% lat (usec) : 1000=2.41% lat (msec) : 2=6.38%, 4=28.42%, 10=33.46%, 20=5.63%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.22%, ctx=5859, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1928KiB/s (1974kB/s), 477KiB/s-493KiB/s (488kB/s-505kB/s), io=23.4MiB (24.6MB), run=12280-12454msec WRITE: bw=2019KiB/s (2067kB/s), 506KiB/s-511KiB/s (518kB/s-523kB/s), io=24.6MiB (25.7MB), run=12280-12454msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31575: Tue Apr 16 17:14:08 2024 read: IOPS=37, BW=149KiB/s (153kB/s)(5992KiB/40168msec) clat (usec): min=1654, max=207873, avg=3447.52, stdev=6138.42 lat (usec): min=1654, max=207875, avg=3448.21, stdev=6138.43 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2212], 60.00th=[ 2376], | 70.00th=[ 2606], 80.00th=[ 3294], 90.00th=[ 6587], 95.00th=[ 9634], | 99.00th=[ 16450], 99.50th=[ 21365], 99.90th=[ 47449], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 40, max= 279, per=25.09%, avg=149.30, stdev=58.96, samples=80 iops : min= 10, max= 69, avg=37.23, stdev=14.70, samples=80 write: IOPS=39, BW=157KiB/s (161kB/s)(6296KiB/40168msec) clat (usec): min=11602, max=68926, avg=22167.24, stdev=7704.84 lat (usec): min=11603, max=68927, avg=22168.19, stdev=7704.85 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13435], 10.00th=[13960], 20.00th=[15270], | 30.00th=[17433], 40.00th=[19530], 50.00th=[21103], 60.00th=[22676], | 70.00th=[23987], 80.00th=[26346], 90.00th=[32113], 95.00th=[37487], | 99.00th=[49021], 99.50th=[53740], 99.90th=[59507], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 48, max= 224, per=25.12%, avg=156.51, stdev=42.41, samples=80 iops : min= 12, max= 56, avg=39.02, stdev=10.59, samples=80 lat (msec) : 2=14.19%, 4=26.43%, 10=5.92%, 20=23.57%, 50=29.46% lat (msec) : 100=0.39%, 250=0.03% cpu : usr=0.09%, sys=4.28%, ctx=3319, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31577: Tue Apr 16 17:14:08 2024 read: IOPS=36, BW=147KiB/s (151kB/s)(5940KiB/40339msec) clat (usec): min=1601, max=175148, avg=3408.35, stdev=5630.45 lat (usec): min=1601, max=175149, avg=3409.01, stdev=5630.46 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1860], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2573], 80.00th=[ 3228], 90.00th=[ 6194], 95.00th=[ 9372], | 99.00th=[ 17171], 99.50th=[ 21627], 99.90th=[ 52167], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 16, max= 288, per=24.73%, avg=147.12, stdev=61.88, samples=80 iops : min= 4, max= 72, avg=36.72, stdev=15.47, samples=80 write: IOPS=39, BW=157KiB/s (161kB/s)(6348KiB/40339msec) clat (usec): min=11718, max=63022, avg=22158.70, stdev=7988.50 lat (usec): min=11719, max=63023, avg=22159.44, stdev=7988.51 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13304], 10.00th=[13960], 20.00th=[15270], | 30.00th=[17171], 40.00th=[19268], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23725], 80.00th=[25822], 90.00th=[32637], 95.00th=[39060], | 99.00th=[50070], 99.50th=[55313], 99.90th=[62129], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 48, max= 208, per=25.14%, avg=156.63, stdev=43.12, samples=80 iops : min= 12, max= 52, avg=39.10, stdev=10.75, samples=80 lat (msec) : 2=14.16%, 4=26.04%, 10=6.05%, 20=24.54%, 50=28.65% lat (msec) : 100=0.52%, 250=0.03% cpu : usr=0.09%, sys=4.23%, ctx=3275, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31579: Tue Apr 16 17:14:08 2024 read: IOPS=37, BW=150KiB/s (153kB/s)(6028KiB/40247msec) clat (usec): min=1587, max=43960, avg=3379.73, stdev=3345.53 lat (usec): min=1588, max=43960, avg=3380.44, stdev=3345.55 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1811], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2343], | 70.00th=[ 2606], 80.00th=[ 3425], 90.00th=[ 6849], 95.00th=[ 9896], | 99.00th=[16450], 99.50th=[22414], 99.90th=[39060], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 32, max= 312, per=25.17%, avg=149.79, stdev=57.42, samples=80 iops : min= 8, max= 78, avg=37.36, stdev=14.34, samples=80 write: IOPS=38, BW=156KiB/s (159kB/s)(6260KiB/40247msec) clat (msec): min=10, max=201, avg=22.39, stdev= 8.90 lat (msec): min=10, max=201, avg=22.39, stdev= 8.90 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 33], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 59], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 48, max= 200, per=24.91%, avg=155.19, stdev=42.46, samples=80 iops : min= 12, max= 50, avg=38.71, stdev=10.65, samples=80 lat (msec) : 2=14.68%, 4=25.59%, 10=6.48%, 20=23.40%, 50=29.49% lat (msec) : 100=0.33%, 250=0.03% cpu : usr=0.06%, sys=4.29%, ctx=3288, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31581: Tue Apr 16 17:14:08 2024 read: IOPS=37, BW=152KiB/s (155kB/s)(6052KiB/39909msec) clat (usec): min=1572, max=52398, avg=3208.12, stdev=3044.53 lat (usec): min=1572, max=52398, avg=3208.79, stdev=3044.54 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2343], | 70.00th=[ 2540], 80.00th=[ 3097], 90.00th=[ 6194], 95.00th=[ 9110], | 99.00th=[13960], 99.50th=[21103], 99.90th=[32900], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 16, max= 303, per=25.37%, avg=150.96, stdev=60.78, samples=79 iops : min= 4, max= 75, avg=37.65, stdev=15.19, samples=79 write: IOPS=39, BW=156KiB/s (160kB/s)(6236KiB/39909msec) clat (msec): min=11, max=190, avg=22.41, stdev= 9.09 lat (msec): min=11, max=190, avg=22.42, stdev= 9.09 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 25], 80.00th=[ 27], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 48], 99.50th=[ 55], 99.90th=[ 67], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 56, max= 208, per=25.00%, avg=155.72, stdev=43.15, samples=79 iops : min= 14, max= 52, avg=38.84, stdev=10.77, samples=79 lat (msec) : 2=14.55%, 4=27.12%, 10=5.66%, 20=23.31%, 50=29.00% lat (msec) : 100=0.33%, 250=0.03% cpu : usr=0.10%, sys=4.36%, ctx=3302, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=595KiB/s (610kB/s), 147KiB/s-152KiB/s (151kB/s-155kB/s), io=23.4MiB (24.6MB), run=39909-40339msec WRITE: bw=623KiB/s (638kB/s), 156KiB/s-157KiB/s (159kB/s-161kB/s), io=24.6MiB (25.7MB), run=39909-40339msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31627: Tue Apr 16 17:14:13 2024 read: IOPS=79, BW=1267KiB/s (1298kB/s)(5920KiB/4671msec) clat (usec): min=53, max=29557, avg=8398.65, stdev=5623.15 lat (usec): min=53, max=29558, avg=8399.20, stdev=5623.16 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 69], 10.00th=[ 75], 20.00th=[ 83], | 30.00th=[ 7242], 40.00th=[ 8848], 50.00th=[ 9634], 60.00th=[10421], | 70.00th=[11076], 80.00th=[12125], 90.00th=[13698], 95.00th=[15795], | 99.00th=[25822], 99.50th=[28705], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 448, max= 1956, per=24.21%, avg=1241.33, stdev=453.27, samples=9 iops : min= 28, max= 122, avg=77.56, stdev=28.28, samples=9 write: IOPS=85, BW=1363KiB/s (1396kB/s)(6368KiB/4671msec) clat (usec): min=656, max=35473, avg=3910.49, stdev=5786.49 lat (usec): min=657, max=35474, avg=3911.39, stdev=5786.52 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 709], 10.00th=[ 734], 20.00th=[ 791], | 30.00th=[ 848], 40.00th=[ 938], 50.00th=[ 1516], 60.00th=[ 2737], | 70.00th=[ 3752], 80.00th=[ 5276], 90.00th=[ 7439], 95.00th=[16712], | 99.00th=[31327], 99.50th=[33162], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 480, max= 2208, per=24.99%, avg=1348.11, stdev=557.98, samples=9 iops : min= 30, max= 138, avg=84.22, stdev=34.82, samples=9 lat (usec) : 100=11.46%, 250=0.52%, 750=7.03%, 1000=15.10% lat (msec) : 2=4.43%, 4=10.29%, 10=24.74%, 20=23.05%, 50=3.39% cpu : usr=0.15%, sys=21.39%, ctx=2029, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31629: Tue Apr 16 17:14:13 2024 read: IOPS=81, BW=1311KiB/s (1342kB/s)(5744KiB/4382msec) clat (usec): min=53, max=28342, avg=8098.07, stdev=5524.80 lat (usec): min=53, max=28343, avg=8098.60, stdev=5524.79 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 69], 10.00th=[ 75], 20.00th=[ 80], | 30.00th=[ 7767], 40.00th=[ 8848], 50.00th=[ 9634], 60.00th=[10421], | 70.00th=[11207], 80.00th=[11994], 90.00th=[13698], 95.00th=[15270], | 99.00th=[20579], 99.50th=[25297], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 448, max= 1920, per=25.04%, avg=1284.00, stdev=449.77, samples=8 iops : min= 28, max= 120, avg=80.25, stdev=28.11, samples=8 write: IOPS=93, BW=1493KiB/s (1529kB/s)(6544KiB/4382msec) clat (usec): min=664, max=42022, avg=3591.47, stdev=5636.73 lat (usec): min=664, max=42024, avg=3592.40, stdev=5636.76 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 832], 40.00th=[ 906], 50.00th=[ 1270], 60.00th=[ 2507], | 70.00th=[ 3359], 80.00th=[ 4817], 90.00th=[ 7439], 95.00th=[14353], | 99.00th=[31589], 99.50th=[34866], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 800, max= 2528, per=26.92%, avg=1452.00, stdev=502.76, samples=8 iops : min= 50, max= 158, avg=90.75, stdev=31.42, samples=8 lat (usec) : 100=11.46%, 250=1.30%, 500=0.13%, 750=5.34%, 1000=18.23% lat (msec) : 2=5.86%, 4=10.94%, 10=22.14%, 20=22.40%, 50=2.21% cpu : usr=0.11%, sys=21.91%, ctx=1923, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31630: Tue Apr 16 17:14:13 2024 read: IOPS=88, BW=1417KiB/s (1451kB/s)(6256KiB/4415msec) clat (usec): min=52, max=29356, avg=7801.42, stdev=5648.15 lat (usec): min=52, max=29357, avg=7801.95, stdev=5648.16 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 68], 10.00th=[ 75], 20.00th=[ 81], | 30.00th=[ 725], 40.00th=[ 8455], 50.00th=[ 9372], 60.00th=[10159], | 70.00th=[11076], 80.00th=[12125], 90.00th=[13698], 95.00th=[15270], | 99.00th=[19792], 99.50th=[26346], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 544, max= 1920, per=25.51%, avg=1308.00, stdev=436.90, samples=8 iops : min= 34, max= 120, avg=81.75, stdev=27.31, samples=8 write: IOPS=85, BW=1366KiB/s (1399kB/s)(6032KiB/4415msec) clat (usec): min=675, max=66847, avg=3598.93, stdev=6610.00 lat (usec): min=675, max=66849, avg=3599.97, stdev=6610.00 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 717], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 840], 40.00th=[ 898], 50.00th=[ 1020], 60.00th=[ 2245], | 70.00th=[ 2966], 80.00th=[ 4883], 90.00th=[ 7373], 95.00th=[12649], | 99.00th=[40109], 99.50th=[47449], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 704, max= 1888, per=24.03%, avg=1296.00, stdev=405.13, samples=8 iops : min= 44, max= 118, avg=81.00, stdev=25.32, samples=8 lat (usec) : 100=13.93%, 250=0.78%, 500=0.13%, 750=5.08%, 1000=19.27% lat (msec) : 2=4.82%, 4=9.24%, 10=22.79%, 20=22.27%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.20%, sys=21.21%, ctx=1935, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31631: Tue Apr 16 17:14:13 2024 read: IOPS=87, BW=1408KiB/s (1441kB/s)(6032KiB/4285msec) clat (usec): min=52, max=29183, avg=8074.77, stdev=5400.28 lat (usec): min=52, max=29184, avg=8075.30, stdev=5400.28 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 68], 10.00th=[ 75], 20.00th=[ 88], | 30.00th=[ 7242], 40.00th=[ 8356], 50.00th=[ 9241], 60.00th=[10028], | 70.00th=[10945], 80.00th=[12256], 90.00th=[13829], 95.00th=[15139], | 99.00th=[20841], 99.50th=[23987], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 928, max= 1920, per=26.99%, avg=1384.00, stdev=323.07, samples=8 iops : min= 58, max= 120, avg=86.50, stdev=20.19, samples=8 write: IOPS=91, BW=1460KiB/s (1495kB/s)(6256KiB/4285msec) clat (usec): min=637, max=41164, avg=3154.22, stdev=4555.53 lat (usec): min=638, max=41166, avg=3155.63, stdev=4555.36 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 807], | 30.00th=[ 865], 40.00th=[ 979], 50.00th=[ 1516], 60.00th=[ 2474], | 70.00th=[ 3195], 80.00th=[ 4228], 90.00th=[ 6128], 95.00th=[10159], | 99.00th=[25035], 99.50th=[38536], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 960, max= 2144, per=27.07%, avg=1460.00, stdev=427.94, samples=8 iops : min= 60, max= 134, avg=91.25, stdev=26.75, samples=8 lat (usec) : 100=10.94%, 250=1.43%, 500=0.26%, 750=3.78%, 1000=17.19% lat (msec) : 2=6.64%, 4=11.59%, 10=25.52%, 20=21.35%, 50=1.30% cpu : usr=0.12%, sys=22.67%, ctx=1980, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5128KiB/s (5251kB/s), 1267KiB/s-1417KiB/s (1298kB/s-1451kB/s), io=23.4MiB (24.5MB), run=4285-4671msec WRITE: bw=5395KiB/s (5524kB/s), 1363KiB/s-1493KiB/s (1396kB/s-1529kB/s), io=24.6MiB (25.8MB), run=4285-4671msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31624: Tue Apr 16 17:14:19 2024 read: IOPS=34, BW=552KiB/s (565kB/s)(5920KiB/10731msec) clat (usec): min=1871, max=27143, avg=4690.76, stdev=3480.60 lat (usec): min=1872, max=27144, avg=4691.39, stdev=3480.61 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 3294], 60.00th=[ 4015], | 70.00th=[ 5080], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[12256], | 99.00th=[18744], 99.50th=[21890], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 287, max= 800, per=24.10%, avg=537.81, stdev=142.37, samples=21 iops : min= 17, max= 50, avg=33.52, stdev= 9.01, samples=21 write: IOPS=37, BW=593KiB/s (608kB/s)(6368KiB/10731msec) clat (usec): min=12429, max=51666, avg=22584.91, stdev=6700.11 lat (usec): min=12431, max=51669, avg=22586.26, stdev=6700.11 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13698], 10.00th=[14746], 20.00th=[16450], | 30.00th=[18482], 40.00th=[20841], 50.00th=[22152], 60.00th=[23200], | 70.00th=[24773], 80.00th=[27132], 90.00th=[31065], 95.00th=[35390], | 99.00th=[43254], 99.50th=[51643], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 416, max= 768, per=25.05%, avg=588.10, stdev=118.74, samples=21 iops : min= 26, max= 48, avg=36.67, stdev= 7.51, samples=21 lat (msec) : 2=0.52%, 4=28.26%, 10=16.02%, 20=21.87%, 50=32.94% lat (msec) : 100=0.39% cpu : usr=0.07%, sys=5.46%, ctx=995, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31625: Tue Apr 16 17:14:19 2024 read: IOPS=33, BW=536KiB/s (549kB/s)(5744KiB/10722msec) clat (usec): min=1823, max=26849, avg=4283.08, stdev=3091.04 lat (usec): min=1823, max=26850, avg=4283.71, stdev=3091.04 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2868], 60.00th=[ 3589], | 70.00th=[ 4621], 80.00th=[ 6063], 90.00th=[ 7504], 95.00th=[11076], | 99.00th=[17171], 99.50th=[19268], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 256, max= 1024, per=24.03%, avg=536.33, stdev=204.19, samples=21 iops : min= 16, max= 64, avg=33.48, stdev=12.82, samples=21 write: IOPS=38, BW=610KiB/s (625kB/s)(6544KiB/10722msec) clat (usec): min=12238, max=49791, avg=22438.17, stdev=6747.03 lat (usec): min=12240, max=49792, avg=22439.66, stdev=6746.96 clat percentiles (usec): | 1.00th=[12911], 5.00th=[14091], 10.00th=[15008], 20.00th=[16319], | 30.00th=[17957], 40.00th=[19268], 50.00th=[21627], 60.00th=[22938], | 70.00th=[25035], 80.00th=[27657], 90.00th=[31589], 95.00th=[35390], | 99.00th=[42206], 99.50th=[43254], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 416, max= 768, per=25.76%, avg=604.86, stdev=113.18, samples=21 iops : min= 26, max= 48, avg=37.76, stdev= 7.11, samples=21 lat (msec) : 2=0.26%, 4=29.82%, 10=13.93%, 20=25.00%, 50=30.99% cpu : usr=0.08%, sys=5.48%, ctx=965, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31626: Tue Apr 16 17:14:19 2024 read: IOPS=38, BW=608KiB/s (623kB/s)(6256KiB/10285msec) clat (usec): min=1956, max=38964, avg=4509.76, stdev=3701.37 lat (usec): min=1957, max=38965, avg=4510.36, stdev=3701.36 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3982], | 70.00th=[ 4817], 80.00th=[ 6063], 90.00th=[ 7701], 95.00th=[11469], | 99.00th=[18482], 99.50th=[30278], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 320, max= 1152, per=27.46%, avg=612.80, stdev=209.27, samples=20 iops : min= 20, max= 72, avg=38.30, stdev=13.08, samples=20 write: IOPS=36, BW=586KiB/s (601kB/s)(6032KiB/10285msec) clat (usec): min=12917, max=66929, avg=22586.08, stdev=6820.94 lat (usec): min=12919, max=66930, avg=22587.45, stdev=6820.93 clat percentiles (usec): | 1.00th=[13304], 5.00th=[14091], 10.00th=[15008], 20.00th=[16581], | 30.00th=[18220], 40.00th=[19792], 50.00th=[21627], 60.00th=[22938], | 70.00th=[25035], 80.00th=[27395], 90.00th=[31851], 95.00th=[35914], | 99.00th=[42206], 99.50th=[45876], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 416, max= 768, per=24.80%, avg=582.40, stdev=117.28, samples=20 iops : min= 26, max= 48, avg=36.40, stdev= 7.33, samples=20 lat (msec) : 2=0.26%, 4=30.34%, 10=16.93%, 20=23.05%, 50=29.30% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=5.71%, ctx=998, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31628: Tue Apr 16 17:14:19 2024 read: IOPS=35, BW=573KiB/s (587kB/s)(6032KiB/10531msec) clat (usec): min=1968, max=23155, avg=4459.86, stdev=3149.09 lat (usec): min=1969, max=23155, avg=4460.64, stdev=3149.05 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 8160], 95.00th=[11731], | 99.00th=[16188], 99.50th=[17957], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 288, max= 832, per=25.46%, avg=568.33, stdev=160.34, samples=21 iops : min= 18, max= 52, avg=35.48, stdev=10.07, samples=21 write: IOPS=37, BW=594KiB/s (608kB/s)(6256KiB/10531msec) clat (usec): min=11945, max=53054, avg=22613.39, stdev=6781.21 lat (usec): min=11947, max=53056, avg=22614.87, stdev=6781.30 clat percentiles (usec): | 1.00th=[12780], 5.00th=[14091], 10.00th=[15008], 20.00th=[16581], | 30.00th=[18220], 40.00th=[19792], 50.00th=[21627], 60.00th=[23200], | 70.00th=[25035], 80.00th=[27395], 90.00th=[31589], 95.00th=[35914], | 99.00th=[41681], 99.50th=[43254], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 384, max= 768, per=25.31%, avg=594.19, stdev=121.32, samples=21 iops : min= 24, max= 48, avg=37.10, stdev= 7.62, samples=21 lat (msec) : 2=0.65%, 4=30.08%, 10=14.97%, 20=24.22%, 50=29.95% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=5.46%, ctx=1007, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2232KiB/s (2286kB/s), 536KiB/s-608KiB/s (549kB/s-623kB/s), io=23.4MiB (24.5MB), run=10285-10731msec WRITE: bw=2348KiB/s (2405kB/s), 586KiB/s-610KiB/s (601kB/s-625kB/s), io=24.6MiB (25.8MB), run=10285-10731msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31644: Tue Apr 16 17:14:26 2024 read: IOPS=1, BW=1311KiB/s (1342kB/s)(7168KiB/5469msec) clat (msec): min=94, max=973, avg=671.48, stdev=335.22 lat (msec): min=94, max=973, avg=671.48, stdev=335.22 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 317], | 30.00th=[ 726], 40.00th=[ 726], 50.00th=[ 785], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 927], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 1689, max= 3373, per=59.83%, avg=2240.20, stdev=651.89, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=0, BW=936KiB/s (959kB/s)(5120KiB/5469msec) clat (msec): min=34, max=274, avg=153.29, stdev=110.45 lat (msec): min=34, max=275, avg=153.35, stdev=110.46 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 95], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 95], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 1689, max= 5060, per=55.94%, avg=2932.33, stdev=1851.34, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 500=25.00%, 750=8.33%, 1000=33.33% cpu : usr=0.00%, sys=30.82%, ctx=1287, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31645: Tue Apr 16 17:14:26 2024 read: IOPS=0, BW=643KiB/s (659kB/s)(2048KiB/3184msec) clat (usec): min=272, max=38364, avg=19318.43, stdev=26934.83 lat (usec): min=273, max=38366, avg=19319.74, stdev=26935.90 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 273], 10.00th=[ 273], 20.00th=[ 273], | 30.00th=[ 273], 40.00th=[ 273], 50.00th=[ 273], 60.00th=[38536], | 70.00th=[38536], 80.00th=[38536], 90.00th=[38536], 95.00th=[38536], | 99.00th=[38536], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 2048, max= 2048, per=54.70%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3216KiB/s (3293kB/s)(10.0MiB/3184msec) clat (msec): min=20, max=2046, avg=314.33, stdev=648.65 lat (msec): min=20, max=2046, avg=314.37, stdev=648.65 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 23], | 30.00th=[ 28], 40.00th=[ 34], 50.00th=[ 43], 60.00th=[ 47], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 751], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 4039, max= 8192, per=100.00%, avg=5442.33, stdev=2381.45, samples=3 iops : min= 3, max= 8, avg= 5.00, stdev= 2.65, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=8.39%, ctx=78, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31646: Tue Apr 16 17:14:26 2024 read: IOPS=1, BW=1964KiB/s (2011kB/s)(6144KiB/3128msec) clat (usec): min=636, max=958070, avg=346084.97, stdev=451368.53 lat (usec): min=637, max=958070, avg=346085.74, stdev=451368.39 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 635], 10.00th=[ 635], 20.00th=[ 53216], | 30.00th=[ 53216], 40.00th=[ 56886], 50.00th=[ 56886], 60.00th=[112722], | 70.00th=[893387], 80.00th=[893387], 90.00th=[960496], 95.00th=[960496], | 99.00th=[960496], 99.50th=[960496], 99.90th=[960496], 99.95th=[960496], | 99.99th=[960496] bw ( KiB/s): min= 2043, max= 8192, per=100.00%, avg=5117.50, stdev=4348.00, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 write: IOPS=1, BW=1964KiB/s (2011kB/s)(6144KiB/3128msec) clat (msec): min=15, max=880, avg=174.90, stdev=346.50 lat (msec): min=15, max=880, avg=174.94, stdev=346.50 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 29], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 2043, max= 4096, per=58.56%, avg=3069.50, stdev=1185.30, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.73, samples=4 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 1000=25.00% cpu : usr=0.00%, sys=11.51%, ctx=383, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31647: Tue Apr 16 17:14:26 2024 read: IOPS=1, BW=1634KiB/s (1673kB/s)(5120KiB/3134msec) clat (usec): min=405, max=792412, avg=177589.77, stdev=345901.90 lat (usec): min=406, max=792413, avg=177591.04, stdev=345901.50 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 408], | 30.00th=[ 930], 40.00th=[ 930], 50.00th=[ 2802], 60.00th=[ 2802], | 70.00th=[ 91751], 80.00th=[ 91751], 90.00th=[792724], 95.00th=[792724], | 99.00th=[792724], 99.50th=[792724], 99.90th=[792724], 99.95th=[792724], | 99.99th=[792724] bw ( KiB/s): min= 4087, max= 4096, per=100.00%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2287KiB/s (2342kB/s)(7168KiB/3134msec) clat (msec): min=27, max=1036, avg=320.60, stdev=434.54 lat (msec): min=27, max=1036, avg=320.65, stdev=434.54 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 45], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 74], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 869], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2043, max= 4096, per=58.54%, avg=3068.50, stdev=1181.27, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 4=8.33%, 50=16.67%, 100=25.00%, 250=8.33%, 1000=16.67% cpu : usr=0.00%, sys=10.88%, ctx=328, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3745KiB/s (3835kB/s), 643KiB/s-1964KiB/s (659kB/s-2011kB/s), io=20.0MiB (20.0MB), run=3128-5469msec WRITE: bw=5243KiB/s (5368kB/s), 936KiB/s-3216KiB/s (959kB/s-3293kB/s), io=28.0MiB (29.4MB), run=3128-5469msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31649: Tue Apr 16 17:14:26 2024 read: IOPS=1, BW=1295KiB/s (1326kB/s)(7168KiB/5534msec) clat (msec): min=23, max=921, avg=442.04, stdev=431.06 lat (msec): min=23, max=921, avg=442.04, stdev=431.06 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 35], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 309], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 894], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 1684, max= 3368, per=60.52%, avg=2239.20, stdev=650.41, samples=5 iops : min= 1, max= 3, avg= 2.00, stdev= 0.71, samples=5 write: IOPS=0, BW=925KiB/s (947kB/s)(5120KiB/5534msec) clat (msec): min=164, max=1029, avg=485.39, stdev=358.34 lat (msec): min=164, max=1029, avg=485.48, stdev=358.34 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 262], 40.00th=[ 262], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 1681, max= 2048, per=34.25%, avg=1774.25, stdev=182.51, samples=4 iops : min= 1, max= 2, avg= 1.25, stdev= 0.50, samples=4 lat (msec) : 50=16.67%, 100=8.33%, 250=8.33%, 500=25.00%, 750=8.33% lat (msec) : 1000=25.00% cpu : usr=0.02%, sys=3.51%, ctx=49, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31650: Tue Apr 16 17:14:26 2024 read: IOPS=0, BW=642KiB/s (658kB/s)(2048KiB/3188msec) clat (usec): min=16331, max=55283, avg=35807.59, stdev=27543.33 lat (usec): min=16332, max=55284, avg=35808.38, stdev=27543.54 clat percentiles (usec): | 1.00th=[16319], 5.00th=[16319], 10.00th=[16319], 20.00th=[16319], | 30.00th=[16319], 40.00th=[16319], 50.00th=[16319], 60.00th=[55313], | 70.00th=[55313], 80.00th=[55313], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 2048, max= 2048, per=55.35%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3212KiB/s (3289kB/s)(10.0MiB/3188msec) clat (msec): min=36, max=1854, avg=311.43, stdev=593.82 lat (msec): min=36, max=1855, avg=311.51, stdev=593.82 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 56], 60.00th=[ 57], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 827], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 6144, max=10240, per=100.00%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=50.00%, 1000=8.33% cpu : usr=0.00%, sys=4.30%, ctx=44, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31651: Tue Apr 16 17:14:26 2024 read: IOPS=1, BW=1902KiB/s (1947kB/s)(6144KiB/3231msec) clat (msec): min=16, max=1012, avg=316.87, stdev=440.64 lat (msec): min=16, max=1012, avg=316.87, stdev=440.64 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 52], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 4096, max= 4096, per=100.00%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1902KiB/s (1947kB/s)(6144KiB/3231msec) clat (msec): min=38, max=927, avg=219.81, stdev=348.52 lat (msec): min=38, max=927, avg=219.84, stdev=348.53 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 92], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 2048, max= 4096, per=65.88%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=3.96%, ctx=56, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31652: Tue Apr 16 17:14:26 2024 read: IOPS=1, BW=1601KiB/s (1639kB/s)(5120KiB/3199msec) clat (msec): min=27, max=839, avg=192.51, stdev=361.81 lat (msec): min=27, max=839, avg=192.51, stdev=361.81 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 38], 80.00th=[ 38], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2048, max= 6131, per=100.00%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2241KiB/s (2294kB/s)(7168KiB/3199msec) clat (msec): min=60, max=923, avg=319.25, stdev=406.96 lat (msec): min=60, max=923, avg=319.31, stdev=406.97 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 74], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 88], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 902], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 2043, max= 6144, per=65.82%, avg=3410.00, stdev=2367.71, samples=3 iops : min= 1, max= 6, avg= 2.67, stdev= 2.89, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=8.33%, 1000=25.00% cpu : usr=0.00%, sys=3.78%, ctx=56, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3701KiB/s (3790kB/s), 642KiB/s-1902KiB/s (658kB/s-1947kB/s), io=20.0MiB (20.0MB), run=3188-5534msec WRITE: bw=5181KiB/s (5305kB/s), 925KiB/s-3212KiB/s (947kB/s-3289kB/s), io=28.0MiB (29.4MB), run=3188-5534msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31665: Tue Apr 16 17:14:32 2024 read: IOPS=0, BW=2729KiB/s (2794kB/s)(12.0MiB/4503msec) clat (msec): min=121, max=2695, avg=1500.77, stdev=1296.86 lat (msec): min=121, max=2695, avg=1500.77, stdev=1296.86 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 1687], 50.00th=[ 1687], 60.00th=[ 1687], | 70.00th=[ 2702], 80.00th=[ 2702], 90.00th=[ 2702], 95.00th=[ 2702], | 99.00th=[ 2702], 99.50th=[ 2702], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=40.36%, ctx=1071, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31667: Tue Apr 16 17:14:32 2024 read: IOPS=0, BW=823KiB/s (843kB/s)(4096KiB/4974msec) clat (nsec): min=129520k, max=129520k, avg=129519828.00, stdev= 0.00 lat (nsec): min=129521k, max=129521k, avg=129520724.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1647KiB/s (1686kB/s)(8192KiB/4974msec) clat (msec): min=306, max=4531, avg=2418.96, stdev=2988.03 lat (msec): min=306, max=4532, avg=2419.25, stdev=2988.09 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 4530], | 70.00th=[ 4530], 80.00th=[ 4530], 90.00th=[ 4530], 95.00th=[ 4530], | 99.00th=[ 4530], 99.50th=[ 4530], 99.90th=[ 4530], 99.95th=[ 4530], | 99.99th=[ 4530] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.02%, sys=4.22%, ctx=316, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31668: Tue Apr 16 17:14:32 2024 write: IOPS=0, BW=2465KiB/s (2525kB/s)(12.0MiB/4984msec) clat (msec): min=98, max=4545, avg=1658.98, stdev=2502.19 lat (msec): min=98, max=4545, avg=1659.27, stdev=2502.22 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 4530], 80.00th=[ 4530], 90.00th=[ 4530], 95.00th=[ 4530], | 99.00th=[ 4530], 99.50th=[ 4530], 99.90th=[ 4530], 99.95th=[ 4530], | 99.99th=[ 4530] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.10%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31669: Tue Apr 16 17:14:32 2024 write: IOPS=0, BW=2541KiB/s (2602kB/s)(12.0MiB/4835msec) clat (msec): min=99, max=4519, avg=1609.06, stdev=2521.28 lat (msec): min=99, max=4519, avg=1609.37, stdev=2521.25 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 4530], 80.00th=[ 4530], 90.00th=[ 4530], 95.00th=[ 4530], | 99.00th=[ 4530], 99.50th=[ 4530], 99.90th=[ 4530], 99.95th=[ 4530], | 99.99th=[ 4530] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=5.25%, ctx=65, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3294KiB/s (3373kB/s), 823KiB/s-2729KiB/s (843kB/s-2794kB/s), io=16.0MiB (16.8MB), run=4503-4974msec WRITE: bw=6575KiB/s (6732kB/s), 1647KiB/s-2541KiB/s (1686kB/s-2602kB/s), io=32.0MiB (33.6MB), run=4835-4984msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31662: Tue Apr 16 17:14:32 2024 read: IOPS=0, BW=2599KiB/s (2661kB/s)(12.0MiB/4728msec) clat (msec): min=156, max=2711, avg=1571.11, stdev=1299.62 lat (msec): min=156, max=2711, avg=1571.11, stdev=1299.62 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 1854], 50.00th=[ 1854], 60.00th=[ 1854], | 70.00th=[ 2702], 80.00th=[ 2702], 90.00th=[ 2702], 95.00th=[ 2702], | 99.00th=[ 2702], 99.50th=[ 2702], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7861, max= 7907, per=100.00%, avg=7884.00, stdev=32.53, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=2.58%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31663: Tue Apr 16 17:14:32 2024 read: IOPS=0, BW=806KiB/s (825kB/s)(4096KiB/5084msec) clat (nsec): min=185505k, max=185505k, avg=185505429.00, stdev= 0.00 lat (nsec): min=185506k, max=185506k, avg=185506476.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1611KiB/s (1650kB/s)(8192KiB/5084msec) clat (msec): min=343, max=4540, avg=2442.08, stdev=2967.34 lat (msec): min=344, max=4540, avg=2442.41, stdev=2967.39 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 4530], | 70.00th=[ 4530], 80.00th=[ 4530], 90.00th=[ 4530], 95.00th=[ 4530], | 99.00th=[ 4530], 99.50th=[ 4530], 99.90th=[ 4530], 99.95th=[ 4530], | 99.99th=[ 4530] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.02%, sys=2.14%, ctx=31, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31664: Tue Apr 16 17:14:32 2024 write: IOPS=8, BW=34.6MiB/s (36.3MB/s)(12.0MiB/347msec) clat (msec): min=66, max=178, avg=115.17, stdev=57.62 lat (msec): min=66, max=179, avg=115.38, stdev=57.71 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 101], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.58%, sys=25.14%, ctx=63, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31666: Tue Apr 16 17:14:32 2024 write: IOPS=6, BW=25.3MiB/s (26.5MB/s)(12.0MiB/474msec) clat (msec): min=99, max=250, avg=156.88, stdev=82.17 lat (msec): min=99, max=251, avg=157.19, stdev=82.13 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=19.66%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3223KiB/s (3300kB/s), 806KiB/s-2599KiB/s (825kB/s-2661kB/s), io=16.0MiB (16.8MB), run=4728-5084msec WRITE: bw=6445KiB/s (6600kB/s), 1611KiB/s-34.6MiB/s (1650kB/s-36.3MB/s), io=32.0MiB (33.6MB), run=347-5084msec