== sanity test 398b: DIO and buffer IO race ============== 23:22:22 (1713410542) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.19302 s, 42.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30041: Wed Apr 17 23:22:36 2024 read: IOPS=136, BW=548KiB/s (561kB/s)(5992KiB/10936msec) clat (usec): min=47, max=40003, avg=3431.40, stdev=3183.12 lat (usec): min=47, max=40003, avg=3432.92, stdev=3182.99 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 84], | 30.00th=[ 2089], 40.00th=[ 2474], 50.00th=[ 2900], 60.00th=[ 3458], | 70.00th=[ 4178], 80.00th=[ 5407], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[12518], 99.50th=[15533], 99.90th=[27395], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 272, max= 808, per=25.54%, avg=547.52, stdev=99.28, samples=21 iops : min= 68, max= 202, avg=136.81, stdev=24.81, samples=21 write: IOPS=143, BW=576KiB/s (590kB/s)(6296KiB/10936msec) clat (usec): min=430, max=122565, avg=3594.18, stdev=5182.04 lat (usec): min=431, max=122565, avg=3595.09, stdev=5182.16 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 594], | 30.00th=[ 676], 40.00th=[ 816], 50.00th=[ 1385], 60.00th=[ 3228], | 70.00th=[ 4490], 80.00th=[ 6325], 90.00th=[ 8979], 95.00th=[ 11076], | 99.00th=[ 16909], 99.50th=[ 20317], 99.90th=[ 65274], 99.95th=[122160], | 99.99th=[122160] bw ( KiB/s): min= 328, max= 792, per=25.47%, avg=571.90, stdev=98.48, samples=21 iops : min= 82, max= 198, avg=142.90, stdev=24.61, samples=21 lat (usec) : 50=0.46%, 100=10.55%, 250=0.88%, 500=1.11%, 750=17.77% lat (usec) : 1000=4.65% lat (msec) : 2=5.86%, 4=25.81%, 10=27.96%, 20=4.52%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.25%, sys=26.31%, ctx=5579, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30043: Wed Apr 17 23:22:36 2024 read: IOPS=132, BW=530KiB/s (543kB/s)(5940KiB/11197msec) clat (usec): min=46, max=161856, avg=3461.31, stdev=5252.86 lat (usec): min=46, max=161857, avg=3462.06, stdev=5252.85 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 70], 20.00th=[ 75], | 30.00th=[ 1860], 40.00th=[ 2376], 50.00th=[ 2868], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5604], 90.00th=[ 7373], 95.00th=[ 9110], | 99.00th=[ 13304], 99.50th=[ 17433], 99.90th=[ 33162], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 16, max= 718, per=24.57%, avg=526.77, stdev=145.98, samples=22 iops : min= 4, max= 179, avg=131.64, stdev=36.45, samples=22 write: IOPS=141, BW=567KiB/s (581kB/s)(6348KiB/11197msec) clat (usec): min=408, max=60029, avg=3727.12, stdev=4708.95 lat (usec): min=408, max=60030, avg=3727.90, stdev=4708.94 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 523], 10.00th=[ 553], 20.00th=[ 603], | 30.00th=[ 676], 40.00th=[ 816], 50.00th=[ 1565], 60.00th=[ 3425], | 70.00th=[ 4621], 80.00th=[ 6259], 90.00th=[ 9110], 95.00th=[12125], | 99.00th=[21365], 99.50th=[29492], 99.90th=[43254], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 56, max= 848, per=24.94%, avg=559.86, stdev=146.83, samples=22 iops : min= 14, max= 212, avg=139.91, stdev=36.69, samples=22 lat (usec) : 50=0.94%, 100=12.17%, 250=0.72%, 500=1.33%, 750=17.51% lat (usec) : 1000=4.30% lat (msec) : 2=5.34%, 4=24.15%, 10=27.99%, 20=4.75%, 50=0.72% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.22%, sys=25.24%, ctx=5351, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30045: Wed Apr 17 23:22:36 2024 read: IOPS=136, BW=546KiB/s (559kB/s)(6028KiB/11040msec) clat (usec): min=46, max=160338, avg=3584.90, stdev=5679.53 lat (usec): min=46, max=160339, avg=3585.68, stdev=5679.40 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 79], | 30.00th=[ 2024], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 5604], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[ 13829], 99.50th=[ 16450], 99.90th=[ 89654], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 264, max= 768, per=24.94%, avg=534.67, stdev=107.73, samples=21 iops : min= 66, max= 192, avg=133.62, stdev=26.89, samples=21 write: IOPS=141, BW=567KiB/s (581kB/s)(6260KiB/11040msec) clat (usec): min=409, max=43869, avg=3512.78, stdev=4119.94 lat (usec): min=410, max=43869, avg=3514.25, stdev=4119.92 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 652], 40.00th=[ 783], 50.00th=[ 1270], 60.00th=[ 3195], | 70.00th=[ 4490], 80.00th=[ 6128], 90.00th=[ 9241], 95.00th=[11338], | 99.00th=[17433], 99.50th=[22676], 99.90th=[25560], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 320, max= 784, per=24.73%, avg=555.24, stdev=122.11, samples=21 iops : min= 80, max= 196, avg=138.76, stdev=30.50, samples=21 lat (usec) : 50=0.62%, 100=11.07%, 250=0.88%, 500=1.76%, 750=18.13% lat (usec) : 1000=4.04% lat (msec) : 2=5.70%, 4=24.71%, 10=27.64%, 20=4.95%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=25.72%, ctx=5480, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30047: Wed Apr 17 23:22:36 2024 read: IOPS=137, BW=552KiB/s (565kB/s)(6052KiB/10968msec) clat (usec): min=47, max=38704, avg=3333.19, stdev=3253.90 lat (usec): min=47, max=38705, avg=3334.00, stdev=3253.80 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 62], 10.00th=[ 70], 20.00th=[ 76], | 30.00th=[ 1975], 40.00th=[ 2409], 50.00th=[ 2868], 60.00th=[ 3392], | 70.00th=[ 4113], 80.00th=[ 5473], 90.00th=[ 7504], 95.00th=[ 8717], | 99.00th=[13304], 99.50th=[14877], 99.90th=[37487], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 112, max= 744, per=25.35%, avg=543.57, stdev=144.00, samples=21 iops : min= 28, max= 186, avg=135.86, stdev=36.03, samples=21 write: IOPS=142, BW=569KiB/s (582kB/s)(6236KiB/10968msec) clat (usec): min=415, max=198790, avg=3706.91, stdev=6882.32 lat (usec): min=415, max=198792, avg=3708.07, stdev=6882.27 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 594], | 30.00th=[ 660], 40.00th=[ 791], 50.00th=[ 1352], 60.00th=[ 3228], | 70.00th=[ 4359], 80.00th=[ 6128], 90.00th=[ 9241], 95.00th=[ 11863], | 99.00th=[ 19006], 99.50th=[ 27132], 99.90th=[ 71828], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 152, max= 736, per=24.94%, avg=559.90, stdev=129.89, samples=21 iops : min= 38, max= 184, avg=139.95, stdev=32.48, samples=21 lat (usec) : 50=0.88%, 100=11.85%, 250=0.85%, 500=1.56%, 750=18.03% lat (usec) : 1000=4.13% lat (msec) : 2=5.34%, 4=24.77%, 10=27.21%, 20=4.85%, 50=0.42% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.16%, sys=25.39%, ctx=5469, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2145KiB/s (2196kB/s), 530KiB/s-552KiB/s (543kB/s-565kB/s), io=23.4MiB (24.6MB), run=10936-11197msec WRITE: bw=2245KiB/s (2299kB/s), 567KiB/s-576KiB/s (581kB/s-590kB/s), io=24.6MiB (25.7MB), run=10936-11197msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30040: Wed Apr 17 23:23:03 2024 read: IOPS=38, BW=156KiB/s (160kB/s)(5992KiB/38412msec) clat (usec): min=1424, max=193199, avg=3434.47, stdev=5923.50 lat (usec): min=1425, max=193200, avg=3435.13, stdev=5923.52 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1893], | 30.00th=[ 1958], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2540], 80.00th=[ 3032], 90.00th=[ 6915], 95.00th=[ 10421], | 99.00th=[ 17171], 99.50th=[ 19792], 99.90th=[ 51119], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 16, max= 280, per=24.98%, avg=155.62, stdev=60.17, samples=76 iops : min= 4, max= 70, avg=38.82, stdev=15.06, samples=76 write: IOPS=40, BW=164KiB/s (168kB/s)(6296KiB/38412msec) clat (msec): min=10, max=106, avg=21.05, stdev= 7.71 lat (msec): min=10, max=106, avg=21.05, stdev= 7.71 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 31], 95.00th=[ 37], | 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 54], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 40, max= 216, per=25.01%, avg=163.30, stdev=46.11, samples=76 iops : min= 10, max= 54, avg=40.74, stdev=11.54, samples=76 lat (msec) : 2=17.61%, 4=23.31%, 10=5.24%, 20=27.90%, 50=25.49% lat (msec) : 100=0.39%, 250=0.07% cpu : usr=0.09%, sys=4.09%, ctx=3331, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30042: Wed Apr 17 23:23:03 2024 read: IOPS=38, BW=154KiB/s (158kB/s)(5940KiB/38484msec) clat (usec): min=1521, max=165014, avg=3398.39, stdev=5519.57 lat (usec): min=1522, max=165016, avg=3399.00, stdev=5519.60 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1762], 10.00th=[ 1811], 20.00th=[ 1893], | 30.00th=[ 1975], 40.00th=[ 2040], 50.00th=[ 2114], 60.00th=[ 2278], | 70.00th=[ 2507], 80.00th=[ 3032], 90.00th=[ 7046], 95.00th=[ 10552], | 99.00th=[ 15533], 99.50th=[ 19006], 99.90th=[ 74974], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 16, max= 312, per=24.73%, avg=154.07, stdev=70.86, samples=76 iops : min= 4, max= 78, avg=38.41, stdev=17.78, samples=76 write: IOPS=41, BW=165KiB/s (169kB/s)(6348KiB/38484msec) clat (msec): min=11, max=118, avg=20.98, stdev= 7.92 lat (msec): min=11, max=118, avg=20.98, stdev= 7.92 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 60], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 56, max= 216, per=25.11%, avg=163.96, stdev=45.63, samples=76 iops : min= 14, max= 54, avg=40.88, stdev=11.49, samples=76 lat (msec) : 2=16.34%, 4=24.87%, 10=4.20%, 20=29.49%, 50=24.74% lat (msec) : 100=0.29%, 250=0.07% cpu : usr=0.06%, sys=4.14%, ctx=3293, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30044: Wed Apr 17 23:23:03 2024 read: IOPS=39, BW=157KiB/s (161kB/s)(6028KiB/38384msec) clat (usec): min=1537, max=40976, avg=3409.95, stdev=3297.50 lat (usec): min=1538, max=40977, avg=3410.55, stdev=3297.52 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1893], | 30.00th=[ 1958], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2540], 80.00th=[ 3228], 90.00th=[ 7439], 95.00th=[11076], | 99.00th=[16188], 99.50th=[19792], 99.90th=[22938], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 40, max= 312, per=25.23%, avg=157.21, stdev=61.81, samples=76 iops : min= 10, max= 78, avg=39.26, stdev=15.46, samples=76 write: IOPS=40, BW=163KiB/s (167kB/s)(6260KiB/38384msec) clat (msec): min=11, max=174, avg=21.15, stdev= 8.98 lat (msec): min=11, max=174, avg=21.15, stdev= 8.98 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 38], | 99.00th=[ 49], 99.50th=[ 55], 99.90th=[ 118], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 48, max= 224, per=24.91%, avg=162.68, stdev=48.42, samples=76 iops : min= 12, max= 56, avg=40.63, stdev=12.15, samples=76 lat (msec) : 2=16.99%, 4=23.54%, 10=5.31%, 20=28.84%, 50=24.87% lat (msec) : 100=0.39%, 250=0.07% cpu : usr=0.09%, sys=4.12%, ctx=3281, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30046: Wed Apr 17 23:23:03 2024 read: IOPS=39, BW=160KiB/s (164kB/s)(6052KiB/37894msec) clat (usec): min=1524, max=40890, avg=3235.20, stdev=3137.42 lat (usec): min=1525, max=40890, avg=3235.84, stdev=3137.41 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1762], 10.00th=[ 1811], 20.00th=[ 1909], | 30.00th=[ 1958], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2507], 80.00th=[ 2966], 90.00th=[ 6325], 95.00th=[10552], | 99.00th=[16057], 99.50th=[18220], 99.90th=[35914], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 32, max= 320, per=25.53%, avg=159.08, stdev=62.31, samples=75 iops : min= 8, max= 80, avg=39.72, stdev=15.63, samples=75 write: IOPS=41, BW=165KiB/s (169kB/s)(6236KiB/37894msec) clat (msec): min=10, max=198, avg=21.08, stdev= 8.93 lat (msec): min=10, max=198, avg=21.08, stdev= 8.93 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 50], 99.50th=[ 54], 99.90th=[ 79], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 40, max= 208, per=25.15%, avg=164.20, stdev=46.29, samples=75 iops : min= 10, max= 52, avg=41.00, stdev=11.65, samples=75 lat (msec) : 2=18.00%, 4=23.63%, 10=4.88%, 20=28.03%, 50=25.07% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.08%, sys=4.18%, ctx=3278, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=624KiB/s (639kB/s), 154KiB/s-160KiB/s (158kB/s-164kB/s), io=23.4MiB (24.6MB), run=37894-38484msec WRITE: bw=653KiB/s (669kB/s), 163KiB/s-165KiB/s (167kB/s-169kB/s), io=24.6MiB (25.7MB), run=37894-38484msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30097: Wed Apr 17 23:23:09 2024 read: IOPS=83, BW=1338KiB/s (1370kB/s)(5920KiB/4424msec) clat (usec): min=51, max=44180, avg=8999.13, stdev=5376.41 lat (usec): min=51, max=44181, avg=8999.92, stdev=5376.06 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 68], 10.00th=[ 75], 20.00th=[ 3851], | 30.00th=[ 8291], 40.00th=[ 8979], 50.00th=[ 9765], 60.00th=[10421], | 70.00th=[11600], 80.00th=[12649], 90.00th=[14353], 95.00th=[15926], | 99.00th=[20579], 99.50th=[27657], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 1056, max= 1628, per=25.35%, avg=1303.12, stdev=209.74, samples=8 iops : min= 66, max= 101, avg=81.25, stdev=12.93, samples=8 write: IOPS=89, BW=1439KiB/s (1474kB/s)(6368KiB/4424msec) clat (usec): min=651, max=19337, avg=2732.85, stdev=2799.14 lat (usec): min=652, max=19338, avg=2733.99, stdev=2799.00 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 717], 10.00th=[ 758], 20.00th=[ 799], | 30.00th=[ 840], 40.00th=[ 906], 50.00th=[ 1270], 60.00th=[ 2638], | 70.00th=[ 3261], 80.00th=[ 4555], 90.00th=[ 5932], 95.00th=[ 7898], | 99.00th=[14615], 99.50th=[19006], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 992, max= 1664, per=26.31%, avg=1423.00, stdev=225.07, samples=8 iops : min= 62, max= 104, avg=88.75, stdev=13.97, samples=8 lat (usec) : 100=8.20%, 250=1.04%, 500=0.13%, 750=4.95%, 1000=18.62% lat (msec) : 2=4.04%, 4=12.63%, 10=26.30%, 20=23.57%, 50=0.52% cpu : usr=0.07%, sys=22.74%, ctx=2019, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30099: Wed Apr 17 23:23:09 2024 read: IOPS=79, BW=1273KiB/s (1304kB/s)(5744KiB/4512msec) clat (usec): min=50, max=29375, avg=8394.70, stdev=5334.43 lat (usec): min=51, max=29377, avg=8395.26, stdev=5334.45 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 64], 10.00th=[ 73], 20.00th=[ 83], | 30.00th=[ 7898], 40.00th=[ 8848], 50.00th=[ 9765], 60.00th=[10421], | 70.00th=[11207], 80.00th=[12256], 90.00th=[14222], 95.00th=[15533], | 99.00th=[18482], 99.50th=[23987], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 224, max= 1664, per=24.47%, avg=1258.00, stdev=407.65, samples=9 iops : min= 14, max= 104, avg=78.44, stdev=25.41, samples=9 write: IOPS=90, BW=1450KiB/s (1485kB/s)(6544KiB/4512msec) clat (usec): min=648, max=113385, avg=3643.82, stdev=7418.30 lat (usec): min=649, max=113387, avg=3647.51, stdev=7417.53 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 840], 40.00th=[ 963], 50.00th=[ 1598], 60.00th=[ 2704], | 70.00th=[ 3261], 80.00th=[ 4752], 90.00th=[ 6783], 95.00th=[ 9634], | 99.00th=[ 30278], 99.50th=[ 32375], 99.90th=[113771], 99.95th=[113771], | 99.99th=[113771] bw ( KiB/s): min= 448, max= 1980, per=26.55%, avg=1435.56, stdev=439.58, samples=9 iops : min= 28, max= 123, avg=89.56, stdev=27.28, samples=9 lat (usec) : 100=10.68%, 250=0.65%, 750=5.47%, 1000=16.54% lat (msec) : 2=5.08%, 4=14.19%, 10=23.31%, 20=22.14%, 50=1.82% lat (msec) : 250=0.13% cpu : usr=0.16%, sys=20.22%, ctx=1872, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30101: Wed Apr 17 23:23:09 2024 read: IOPS=83, BW=1343KiB/s (1375kB/s)(6256KiB/4659msec) clat (usec): min=49, max=45186, avg=8909.00, stdev=5249.60 lat (usec): min=50, max=45187, avg=8909.56, stdev=5249.61 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 72], 10.00th=[ 76], 20.00th=[ 6390], | 30.00th=[ 8160], 40.00th=[ 9110], 50.00th=[ 9896], 60.00th=[10552], | 70.00th=[11469], 80.00th=[12256], 90.00th=[13960], 95.00th=[15401], | 99.00th=[19530], 99.50th=[25560], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 864, max= 1824, per=25.92%, avg=1332.44, stdev=280.48, samples=9 iops : min= 54, max= 114, avg=83.11, stdev=17.39, samples=9 write: IOPS=80, BW=1295KiB/s (1326kB/s)(6032KiB/4659msec) clat (usec): min=643, max=46995, avg=3098.86, stdev=4632.49 lat (usec): min=644, max=46997, avg=3099.79, stdev=4632.51 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 783], | 30.00th=[ 816], 40.00th=[ 898], 50.00th=[ 1172], 60.00th=[ 2606], | 70.00th=[ 3359], 80.00th=[ 4621], 90.00th=[ 6325], 95.00th=[ 8094], | 99.00th=[28181], 99.50th=[35914], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 1021, max= 1692, per=23.19%, avg=1254.33, stdev=216.88, samples=9 iops : min= 63, max= 105, avg=78.22, stdev=13.48, samples=9 lat (usec) : 50=0.13%, 100=9.24%, 250=0.13%, 500=0.26%, 750=5.34% lat (usec) : 1000=17.58% lat (msec) : 2=4.04%, 4=10.03%, 10=27.08%, 20=25.00%, 50=1.17% cpu : usr=0.15%, sys=20.78%, ctx=2111, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30102: Wed Apr 17 23:23:09 2024 read: IOPS=81, BW=1303KiB/s (1334kB/s)(6032KiB/4630msec) clat (usec): min=50, max=37140, avg=8768.45, stdev=5303.84 lat (usec): min=50, max=37141, avg=8769.01, stdev=5303.85 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 64], 10.00th=[ 74], 20.00th=[ 97], | 30.00th=[ 8160], 40.00th=[ 8979], 50.00th=[ 9765], 60.00th=[10552], | 70.00th=[11338], 80.00th=[12518], 90.00th=[14091], 95.00th=[15664], | 99.00th=[19530], 99.50th=[26608], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 416, max= 1536, per=24.90%, avg=1280.00, stdev=339.41, samples=9 iops : min= 26, max= 96, avg=80.00, stdev=21.21, samples=9 write: IOPS=84, BW=1351KiB/s (1384kB/s)(6256KiB/4630msec) clat (usec): min=664, max=49929, avg=3364.01, stdev=5638.39 lat (usec): min=664, max=49930, avg=3364.92, stdev=5638.58 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 848], 40.00th=[ 938], 50.00th=[ 1237], 60.00th=[ 2638], | 70.00th=[ 3130], 80.00th=[ 4359], 90.00th=[ 6325], 95.00th=[ 9372], | 99.00th=[35390], 99.50th=[44827], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 480, max= 1760, per=24.79%, avg=1340.44, stdev=360.42, samples=9 iops : min= 30, max= 110, avg=83.78, stdev=22.53, samples=9 lat (usec) : 100=9.90%, 250=0.52%, 750=4.43%, 1000=18.62% lat (msec) : 2=3.78%, 4=13.15%, 10=24.09%, 20=23.83%, 50=1.69% cpu : usr=0.11%, sys=20.54%, ctx=1970, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5141KiB/s (5264kB/s), 1273KiB/s-1343KiB/s (1304kB/s-1375kB/s), io=23.4MiB (24.5MB), run=4424-4659msec WRITE: bw=5409KiB/s (5539kB/s), 1295KiB/s-1450KiB/s (1326kB/s-1485kB/s), io=24.6MiB (25.8MB), run=4424-4659msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30095: Wed Apr 17 23:23:14 2024 read: IOPS=37, BW=593KiB/s (607kB/s)(5920KiB/9985msec) clat (usec): min=1941, max=44161, avg=4317.29, stdev=4445.05 lat (usec): min=1941, max=44162, avg=4317.90, stdev=4445.06 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2245], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2606], 60.00th=[ 2933], | 70.00th=[ 3294], 80.00th=[ 5342], 90.00th=[ 9110], 95.00th=[12125], | 99.00th=[23462], 99.50th=[39060], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 352, max= 896, per=24.67%, avg=587.79, stdev=158.29, samples=19 iops : min= 22, max= 56, avg=36.74, stdev= 9.89, samples=19 write: IOPS=39, BW=638KiB/s (653kB/s)(6368KiB/9985msec) clat (usec): min=11133, max=44392, avg=21054.94, stdev=5938.10 lat (usec): min=11134, max=44394, avg=21056.29, stdev=5938.00 clat percentiles (usec): | 1.00th=[11863], 5.00th=[12780], 10.00th=[13960], 20.00th=[16057], | 30.00th=[17957], 40.00th=[19268], 50.00th=[20841], 60.00th=[21890], | 70.00th=[22938], 80.00th=[25035], 90.00th=[28181], 95.00th=[31851], | 99.00th=[42206], 99.50th=[43779], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 416, max= 800, per=25.06%, avg=628.21, stdev=137.52, samples=19 iops : min= 26, max= 50, avg=39.26, stdev= 8.59, samples=19 lat (msec) : 2=0.26%, 4=36.46%, 10=7.03%, 20=27.34%, 50=28.91% cpu : usr=0.05%, sys=5.11%, ctx=847, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30096: Wed Apr 17 23:23:14 2024 read: IOPS=35, BW=572KiB/s (585kB/s)(5744KiB/10048msec) clat (usec): min=1880, max=32185, avg=3802.23, stdev=3429.61 lat (usec): min=1881, max=32185, avg=3802.82, stdev=3429.61 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2089], 10.00th=[ 2147], 20.00th=[ 2245], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2606], 60.00th=[ 2802], | 70.00th=[ 3195], 80.00th=[ 4015], 90.00th=[ 6915], 95.00th=[10552], | 99.00th=[17695], 99.50th=[28967], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 224, max= 1120, per=23.82%, avg=567.58, stdev=236.08, samples=19 iops : min= 14, max= 70, avg=35.47, stdev=14.75, samples=19 write: IOPS=40, BW=651KiB/s (667kB/s)(6544KiB/10048msec) clat (msec): min=11, max=109, avg=21.21, stdev= 7.97 lat (msec): min=11, max=109, avg=21.21, stdev= 7.97 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 31], 95.00th=[ 34], | 99.00th=[ 43], 99.50th=[ 45], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 448, max= 896, per=25.60%, avg=641.68, stdev=150.27, samples=19 iops : min= 28, max= 56, avg=40.11, stdev= 9.39, samples=19 lat (msec) : 2=0.52%, 4=36.85%, 10=6.77%, 20=28.91%, 50=26.82% lat (msec) : 250=0.13% cpu : usr=0.03%, sys=4.95%, ctx=856, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30098: Wed Apr 17 23:23:14 2024 read: IOPS=40, BW=647KiB/s (662kB/s)(6256KiB/9672msec) clat (usec): min=1949, max=24243, avg=4002.74, stdev=3457.70 lat (usec): min=1950, max=24243, avg=4003.32, stdev=3457.70 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2147], 10.00th=[ 2180], 20.00th=[ 2245], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2638], 60.00th=[ 2900], | 70.00th=[ 3228], 80.00th=[ 4817], 90.00th=[ 7635], 95.00th=[12649], | 99.00th=[19268], 99.50th=[23462], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 320, max= 1248, per=27.49%, avg=655.16, stdev=259.83, samples=19 iops : min= 20, max= 78, avg=40.95, stdev=16.24, samples=19 write: IOPS=38, BW=624KiB/s (639kB/s)(6032KiB/9672msec) clat (usec): min=10331, max=51137, avg=21480.88, stdev=7238.70 lat (usec): min=10332, max=51138, avg=21482.19, stdev=7238.62 clat percentiles (usec): | 1.00th=[11863], 5.00th=[12649], 10.00th=[13304], 20.00th=[14877], | 30.00th=[17171], 40.00th=[19268], 50.00th=[20579], 60.00th=[21627], | 70.00th=[23200], 80.00th=[26084], 90.00th=[30802], 95.00th=[34866], | 99.00th=[47973], 99.50th=[50070], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 384, max= 800, per=24.72%, avg=619.79, stdev=157.93, samples=19 iops : min= 24, max= 50, avg=38.74, stdev= 9.87, samples=19 lat (msec) : 2=0.26%, 4=39.06%, 10=7.94%, 20=25.78%, 50=26.69% lat (msec) : 100=0.26% cpu : usr=0.06%, sys=5.36%, ctx=877, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30100: Wed Apr 17 23:23:14 2024 read: IOPS=38, BW=608KiB/s (623kB/s)(6032KiB/9914msec) clat (usec): min=1914, max=29338, avg=3943.61, stdev=3549.03 lat (usec): min=1915, max=29339, avg=3944.23, stdev=3549.00 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3097], 80.00th=[ 4359], 90.00th=[ 7570], 95.00th=[12387], | 99.00th=[18482], 99.50th=[21627], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 320, max= 864, per=25.16%, avg=599.58, stdev=167.25, samples=19 iops : min= 20, max= 54, avg=37.47, stdev=10.45, samples=19 write: IOPS=39, BW=631KiB/s (646kB/s)(6256KiB/9914msec) clat (usec): min=11349, max=60145, avg=21535.37, stdev=6797.55 lat (usec): min=11350, max=60147, avg=21536.64, stdev=6797.57 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12780], 10.00th=[13566], 20.00th=[15401], | 30.00th=[17695], 40.00th=[19268], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23462], 80.00th=[26084], 90.00th=[30802], 95.00th=[34866], | 99.00th=[41157], 99.50th=[45351], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 384, max= 800, per=24.92%, avg=624.84, stdev=153.23, samples=19 iops : min= 24, max= 50, avg=39.05, stdev= 9.58, samples=19 lat (msec) : 2=1.04%, 4=37.24%, 10=6.77%, 20=26.95%, 50=27.86% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=5.02%, ctx=859, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2384KiB/s (2441kB/s), 572KiB/s-647KiB/s (585kB/s-662kB/s), io=23.4MiB (24.5MB), run=9672-10048msec WRITE: bw=2508KiB/s (2568kB/s), 624KiB/s-651KiB/s (639kB/s-667kB/s), io=24.6MiB (25.8MB), run=9672-10048msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30116: Wed Apr 17 23:23:20 2024 read: IOPS=2, BW=2697KiB/s (2761kB/s)(7168KiB/2658msec) clat (usec): min=342, max=1038.3k, avg=345042.30, stdev=440615.99 lat (usec): min=343, max=1038.3k, avg=345043.61, stdev=440616.01 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 343], 10.00th=[ 343], | 20.00th=[ 49546], 30.00th=[ 84411], 40.00th=[ 84411], | 50.00th=[ 108528], 60.00th=[ 208667], 70.00th=[ 208667], | 80.00th=[ 926942], 90.00th=[1035994], 95.00th=[1035994], | 99.00th=[1035994], 99.50th=[1035994], 99.90th=[1035994], | 99.95th=[1035994], 99.99th=[1035994] bw ( KiB/s): min= 2043, max= 5342, per=70.55%, avg=3144.33, stdev=1903.24, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 write: IOPS=1, BW=1926KiB/s (1972kB/s)(5120KiB/2658msec) clat (msec): min=17, max=103, avg=48.08, stdev=35.49 lat (msec): min=17, max=103, avg=48.13, stdev=35.48 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 47], 60.00th=[ 47], | 70.00th=[ 55], 80.00th=[ 55], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 7123, max= 7123, per=100.00%, avg=7123.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=16.67%, 100=16.67%, 250=25.00%, 1000=8.33% cpu : usr=0.08%, sys=26.31%, ctx=722, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30118: Wed Apr 17 23:23:20 2024 read: IOPS=0, BW=819KiB/s (839kB/s)(2048KiB/2500msec) clat (msec): min=50, max=1070, avg=560.52, stdev=721.22 lat (msec): min=50, max=1070, avg=560.52, stdev=721.22 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 1070], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 2048, max= 2048, per=45.95%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4096KiB/s (4194kB/s)(10.0MiB/2500msec) clat (msec): min=20, max=1034, avg=136.71, stdev=315.77 lat (msec): min=20, max=1034, avg=136.77, stdev=315.76 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 29], 50.00th=[ 32], 60.00th=[ 41], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 70], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=19.01%, ctx=378, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30119: Wed Apr 17 23:23:20 2024 read: IOPS=1, BW=1338KiB/s (1370kB/s)(6144KiB/4593msec) clat (usec): min=372, max=972343, avg=350144.10, stdev=470620.80 lat (usec): min=373, max=972343, avg=350144.81, stdev=470620.83 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 783], | 30.00th=[ 783], 40.00th=[ 61080], 50.00th=[ 61080], 60.00th=[129500], | 70.00th=[935330], 80.00th=[935330], 90.00th=[968885], 95.00th=[968885], | 99.00th=[968885], 99.50th=[968885], 99.90th=[968885], 99.95th=[968885], | 99.99th=[968885] bw ( KiB/s): min= 2048, max= 6131, per=76.49%, avg=3409.00, stdev=2357.32, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 write: IOPS=1, BW=1338KiB/s (1370kB/s)(6144KiB/4593msec) clat (msec): min=16, max=2274, avg=413.21, stdev=912.47 lat (msec): min=16, max=2274, avg=413.25, stdev=912.46 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 22], 60.00th=[ 47], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 2048, max= 4096, per=49.19%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 20=16.67%, 50=16.67%, 100=8.33%, 250=16.67%, 1000=16.67% cpu : usr=0.00%, sys=12.22%, ctx=624, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30121: Wed Apr 17 23:23:20 2024 read: IOPS=1, BW=1114KiB/s (1141kB/s)(5120KiB/4594msec) clat (usec): min=364, max=956742, avg=206625.70, stdev=420574.98 lat (usec): min=365, max=956743, avg=206626.37, stdev=420575.33 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 367], | 30.00th=[ 404], 40.00th=[ 404], 50.00th=[ 465], 60.00th=[ 465], | 70.00th=[ 74974], 80.00th=[ 74974], 90.00th=[960496], 95.00th=[960496], | 99.00th=[960496], 99.50th=[960496], 99.90th=[960496], 99.95th=[960496], | 99.99th=[960496] bw ( KiB/s): min= 4087, max= 4087, per=91.70%, avg=4087.00, stdev= 0.00, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1560KiB/s (1598kB/s)(7168KiB/4594msec) clat (msec): min=33, max=2263, avg=506.73, stdev=855.76 lat (msec): min=33, max=2263, avg=506.76, stdev=855.76 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 51], 60.00th=[ 97], | 70.00th=[ 97], 80.00th=[ 1028], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 2048, max= 6131, per=57.34%, avg=3578.50, stdev=1954.37, samples=4 iops : min= 2, max= 5, avg= 3.00, stdev= 1.41, samples=4 lat (usec) : 500=25.00% lat (msec) : 50=25.00%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=6.42%, ctx=320, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4458KiB/s (4565kB/s), 819KiB/s-2697KiB/s (839kB/s-2761kB/s), io=20.0MiB (20.0MB), run=2500-4594msec WRITE: bw=6241KiB/s (6391kB/s), 1338KiB/s-4096KiB/s (1370kB/s-4194kB/s), io=28.0MiB (29.4MB), run=2500-4594msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30117: Wed Apr 17 23:23:20 2024 read: IOPS=1, BW=1503KiB/s (1539kB/s)(7168KiB/4769msec) clat (msec): min=21, max=1038, avg=321.17, stdev=455.22 lat (msec): min=21, max=1038, avg=321.17, stdev=455.22 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 26], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 71], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 927], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2048, max= 6144, per=79.66%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1074KiB/s (1099kB/s)(5120KiB/4769msec) clat (msec): min=29, max=2030, avg=502.85, stdev=855.90 lat (msec): min=29, max=2030, avg=502.90, stdev=855.92 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 122], 40.00th=[ 122], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 2039], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 2048, max= 4096, per=51.21%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=8.33%, 250=33.33%, 1000=8.33% cpu : usr=0.02%, sys=2.96%, ctx=62, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30120: Wed Apr 17 23:23:20 2024 read: IOPS=0, BW=716KiB/s (734kB/s)(2048KiB/2859msec) clat (msec): min=68, max=1021, avg=544.56, stdev=673.86 lat (msec): min=68, max=1021, avg=544.56, stdev=673.86 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 1020], | 70.00th=[ 1020], 80.00th=[ 1020], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 2048, max= 2048, per=47.79%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3582KiB/s (3668kB/s)(10.0MiB/2859msec) clat (msec): min=55, max=1105, avg=176.33, stdev=327.20 lat (msec): min=55, max=1105, avg=176.42, stdev=327.18 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 58], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 68], 60.00th=[ 68], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 118], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67%, 250=16.67% cpu : usr=0.03%, sys=5.74%, ctx=49, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30122: Wed Apr 17 23:23:20 2024 read: IOPS=1, BW=1286KiB/s (1316kB/s)(6144KiB/4779msec) clat (msec): min=19, max=1769, avg=351.04, stdev=698.77 lat (msec): min=19, max=1769, avg=351.04, stdev=698.77 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 45], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1286KiB/s (1316kB/s)(6144KiB/4779msec) clat (msec): min=46, max=2252, avg=444.41, stdev=887.25 lat (msec): min=46, max=2252, avg=444.46, stdev=887.27 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 2048, max= 4096, per=56.90%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=2.70%, ctx=35, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30123: Wed Apr 17 23:23:20 2024 read: IOPS=1, BW=1074KiB/s (1100kB/s)(5120KiB/4767msec) clat (usec): min=24131, max=37490, avg=27721.79, stdev=5656.29 lat (usec): min=24132, max=37491, avg=27722.82, stdev=5656.49 clat percentiles (usec): | 1.00th=[24249], 5.00th=[24249], 10.00th=[24249], 20.00th=[24249], | 30.00th=[24249], 40.00th=[24249], 50.00th=[25035], 60.00th=[25035], | 70.00th=[27657], 80.00th=[27657], 90.00th=[37487], 95.00th=[37487], | 99.00th=[37487], 99.50th=[37487], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1504KiB/s (1540kB/s)(7168KiB/4767msec) clat (msec): min=57, max=2278, avg=660.11, stdev=921.04 lat (msec): min=57, max=2278, avg=660.17, stdev=921.05 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 59], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 169], 60.00th=[ 264], | 70.00th=[ 264], 80.00th=[ 1687], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 2048, max= 6144, per=51.21%, avg=3072.00, stdev=2048.00, samples=4 iops : min= 2, max= 6, avg= 3.00, stdev= 2.00, samples=4 lat (msec) : 50=41.67%, 100=16.67%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=2.81%, ctx=45, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4285KiB/s (4388kB/s), 716KiB/s-1503KiB/s (734kB/s-1539kB/s), io=20.0MiB (20.0MB), run=2859-4779msec WRITE: bw=6000KiB/s (6144kB/s), 1074KiB/s-3582KiB/s (1099kB/s-3668kB/s), io=28.0MiB (29.4MB), run=2859-4779msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30136: Wed Apr 17 23:23:24 2024 read: IOPS=1, BW=4096KiB/s (4194kB/s)(12.0MiB/3000msec) clat (msec): min=91, max=1780, avg=999.61, stdev=851.45 lat (msec): min=91, max=1780, avg=999.61, stdev=851.45 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 1133], 50.00th=[ 1133], 60.00th=[ 1133], | 70.00th=[ 1787], 80.00th=[ 1787], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33% cpu : usr=0.00%, sys=44.71%, ctx=789, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30137: Wed Apr 17 23:23:24 2024 read: IOPS=0, BW=1634KiB/s (1674kB/s)(4096KiB/2506msec) clat (nsec): min=106294k, max=106294k, avg=106293912.00, stdev= 0.00 lat (nsec): min=106295k, max=106295k, avg=106294669.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3269KiB/s (3347kB/s)(8192KiB/2506msec) clat (msec): min=204, max=2193, avg=1199.10, stdev=1406.31 lat (msec): min=205, max=2193, avg=1199.39, stdev=1406.25 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 2198], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8192, max= 8192, per=81.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=7.43%, ctx=221, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30138: Wed Apr 17 23:23:24 2024 write: IOPS=0, BW=3779KiB/s (3869kB/s)(12.0MiB/3252msec) clat (msec): min=374, max=2189, avg=1083.52, stdev=970.78 lat (msec): min=374, max=2190, avg=1083.71, stdev=970.85 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 376], 10.00th=[ 376], 20.00th=[ 376], | 30.00th=[ 376], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7876, max= 8192, per=79.73%, avg=8034.00, stdev=223.45, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.07%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30139: Wed Apr 17 23:23:24 2024 write: IOPS=1, BW=4824KiB/s (4940kB/s)(12.0MiB/2547msec) clat (msec): min=110, max=2186, avg=845.96, stdev=1162.77 lat (msec): min=110, max=2186, avg=846.20, stdev=1162.74 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8175, max= 8192, per=81.22%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=9.90%, ctx=69, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5461KiB/s (5592kB/s), 1634KiB/s-4096KiB/s (1674kB/s-4194kB/s), io=16.0MiB (16.8MB), run=2506-3000msec WRITE: bw=9.84MiB/s (10.3MB/s), 3269KiB/s-4824KiB/s (3347kB/s-4940kB/s), io=32.0MiB (33.6MB), run=2506-3252msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30132: Wed Apr 17 23:23:24 2024 read: IOPS=0, BW=4014KiB/s (4111kB/s)(12.0MiB/3061msec) clat (msec): min=129, max=1784, avg=1019.85, stdev=834.81 lat (msec): min=129, max=1784, avg=1019.85, stdev=834.81 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 1150], 50.00th=[ 1150], 60.00th=[ 1150], | 70.00th=[ 1787], 80.00th=[ 1787], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=3.79%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30133: Wed Apr 17 23:23:24 2024 read: IOPS=0, BW=1525KiB/s (1562kB/s)(4096KiB/2686msec) clat (nsec): min=170069k, max=170069k, avg=170068871.00, stdev= 0.00 lat (nsec): min=170071k, max=170071k, avg=170070549.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3050KiB/s (3123kB/s)(8192KiB/2686msec) clat (msec): min=257, max=2256, avg=1256.98, stdev=1413.30 lat (msec): min=257, max=2256, avg=1257.23, stdev=1413.16 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 2265], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 8192, max= 8192, per=84.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.07%, sys=3.80%, ctx=110, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30134: Wed Apr 17 23:23:24 2024 write: IOPS=0, BW=3623KiB/s (3710kB/s)(12.0MiB/3392msec) clat (msec): min=523, max=2235, avg=1125.03, stdev=962.42 lat (msec): min=524, max=2235, avg=1125.43, stdev=962.46 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 617], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7684, max= 8192, per=82.17%, avg=7938.00, stdev=359.21, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=66.67% cpu : usr=0.00%, sys=4.95%, ctx=27, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30135: Wed Apr 17 23:23:24 2024 write: IOPS=1, BW=5046KiB/s (5168kB/s)(12.0MiB/2435msec) clat (msec): min=105, max=2168, avg=810.29, stdev=1176.50 lat (msec): min=105, max=2168, avg=810.64, stdev=1176.44 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8175, max= 8175, per=84.63%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=6.57%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5352KiB/s (5481kB/s), 1525KiB/s-4014KiB/s (1562kB/s-4111kB/s), io=16.0MiB (16.8MB), run=2686-3061msec WRITE: bw=9660KiB/s (9892kB/s), 3050KiB/s-5046KiB/s (3123kB/s-5168kB/s), io=32.0MiB (33.6MB), run=2435-3392msec