== sanity test 398b: DIO and buffer IO race ============== 23:33:42 (1713411222) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.06878 s, 47.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11706: Wed Apr 17 23:33:56 2024 read: IOPS=131, BW=526KiB/s (538kB/s)(5992KiB/11399msec) clat (usec): min=48, max=146780, avg=4311.11, stdev=4302.24 lat (usec): min=48, max=146780, avg=4311.83, stdev=4302.31 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1860], 10.00th=[ 2147], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 4424], | 70.00th=[ 5014], 80.00th=[ 5669], 90.00th=[ 6783], 95.00th=[ 8291], | 99.00th=[ 11600], 99.50th=[ 13698], 99.90th=[ 20317], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 344, max= 672, per=25.42%, avg=521.64, stdev=82.42, samples=22 iops : min= 86, max= 168, avg=130.36, stdev=20.58, samples=22 write: IOPS=138, BW=552KiB/s (566kB/s)(6296KiB/11399msec) clat (usec): min=425, max=22643, avg=3108.14, stdev=2657.88 lat (usec): min=425, max=22643, avg=3109.25, stdev=2659.13 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 816], 40.00th=[ 2024], 50.00th=[ 2540], 60.00th=[ 3163], | 70.00th=[ 4047], 80.00th=[ 5145], 90.00th=[ 6783], 95.00th=[ 8455], | 99.00th=[11338], 99.50th=[12125], 99.90th=[14746], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 408, max= 712, per=25.42%, avg=546.00, stdev=76.29, samples=22 iops : min= 102, max= 178, avg=136.45, stdev=19.06, samples=22 lat (usec) : 50=0.03%, 100=1.66%, 250=0.20%, 500=1.17%, 750=13.51% lat (usec) : 1000=2.41% lat (msec) : 2=4.49%, 4=37.57%, 10=36.95%, 20=1.92%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=23.73%, ctx=6153, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11707: Wed Apr 17 23:33:56 2024 read: IOPS=130, BW=522KiB/s (535kB/s)(5940KiB/11371msec) clat (usec): min=48, max=20474, avg=4234.99, stdev=2237.34 lat (usec): min=48, max=20475, avg=4236.05, stdev=2237.27 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 1811], 10.00th=[ 2180], 20.00th=[ 2573], | 30.00th=[ 2933], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 4424], | 70.00th=[ 4948], 80.00th=[ 5800], 90.00th=[ 6915], 95.00th=[ 7963], | 99.00th=[11994], 99.50th=[12911], 99.90th=[16188], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 360, max= 696, per=25.27%, avg=518.45, stdev=71.37, samples=22 iops : min= 90, max= 174, avg=129.59, stdev=17.83, samples=22 write: IOPS=139, BW=558KiB/s (572kB/s)(6348KiB/11371msec) clat (usec): min=392, max=145053, avg=3166.75, stdev=4458.79 lat (usec): min=392, max=145054, avg=3168.07, stdev=4458.52 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 775], 40.00th=[ 1991], 50.00th=[ 2573], 60.00th=[ 3097], | 70.00th=[ 3916], 80.00th=[ 5080], 90.00th=[ 6849], 95.00th=[ 8225], | 99.00th=[ 11600], 99.50th=[ 13698], 99.90th=[ 20317], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 392, max= 712, per=25.44%, avg=546.45, stdev=85.71, samples=22 iops : min= 98, max= 178, avg=136.59, stdev=21.44, samples=22 lat (usec) : 50=0.10%, 100=1.76%, 250=0.16%, 500=1.14%, 750=14.13% lat (usec) : 1000=2.18% lat (msec) : 2=4.17%, 4=38.38%, 10=35.77%, 20=2.12%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.89%, ctx=6098, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11708: Wed Apr 17 23:33:56 2024 read: IOPS=129, BW=517KiB/s (530kB/s)(6028KiB/11655msec) clat (usec): min=49, max=19413, avg=4253.90, stdev=2231.33 lat (usec): min=50, max=19413, avg=4254.46, stdev=2231.34 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 1811], 10.00th=[ 2114], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3392], 50.00th=[ 3916], 60.00th=[ 4490], | 70.00th=[ 5080], 80.00th=[ 5735], 90.00th=[ 6980], 95.00th=[ 8291], | 99.00th=[11600], 99.50th=[12780], 99.90th=[15926], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 176, max= 672, per=25.03%, avg=513.65, stdev=98.47, samples=23 iops : min= 44, max= 168, avg=128.39, stdev=24.59, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6260KiB/11655msec) clat (usec): min=414, max=154455, avg=3320.20, stdev=4798.42 lat (usec): min=415, max=154456, avg=3320.87, stdev=4798.44 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 578], | 30.00th=[ 791], 40.00th=[ 2024], 50.00th=[ 2606], 60.00th=[ 3228], | 70.00th=[ 4178], 80.00th=[ 5342], 90.00th=[ 7111], 95.00th=[ 8848], | 99.00th=[ 12649], 99.50th=[ 14746], 99.90th=[ 22676], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 224, max= 720, per=24.80%, avg=532.78, stdev=106.48, samples=23 iops : min= 56, max= 180, avg=133.17, stdev=26.63, samples=23 lat (usec) : 50=0.03%, 100=1.86%, 250=0.10%, 500=1.43%, 750=13.35% lat (usec) : 1000=2.38% lat (msec) : 2=4.95%, 4=36.00%, 10=37.01%, 20=2.80%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=23.49%, ctx=5944, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11710: Wed Apr 17 23:33:56 2024 read: IOPS=129, BW=517KiB/s (530kB/s)(6052KiB/11699msec) clat (usec): min=48, max=19124, avg=4242.98, stdev=2194.26 lat (usec): min=49, max=19124, avg=4243.58, stdev=2194.24 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1942], 10.00th=[ 2180], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 6915], 95.00th=[ 8160], | 99.00th=[11731], 99.50th=[13304], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 208, max= 664, per=25.03%, avg=513.57, stdev=95.51, samples=23 iops : min= 52, max= 166, avg=128.35, stdev=23.85, samples=23 write: IOPS=133, BW=533KiB/s (546kB/s)(6236KiB/11699msec) clat (usec): min=462, max=153728, avg=3350.60, stdev=4798.44 lat (usec): min=462, max=153730, avg=3353.21, stdev=4799.52 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 758], 40.00th=[ 2040], 50.00th=[ 2671], 60.00th=[ 3228], | 70.00th=[ 4047], 80.00th=[ 5473], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[ 12256], 99.50th=[ 16057], 99.90th=[ 24249], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 224, max= 720, per=24.59%, avg=528.17, stdev=98.97, samples=23 iops : min= 56, max= 180, avg=132.00, stdev=24.74, samples=23 lat (usec) : 50=0.03%, 100=1.63%, 250=0.10%, 500=1.17%, 750=13.93% lat (usec) : 1000=2.05% lat (msec) : 2=3.84%, 4=38.61%, 10=36.20%, 20=2.38%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.57%, ctx=6041, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2052KiB/s (2102kB/s), 517KiB/s-526KiB/s (530kB/s-538kB/s), io=23.4MiB (24.6MB), run=11371-11699msec WRITE: bw=2149KiB/s (2200kB/s), 533KiB/s-558KiB/s (546kB/s-572kB/s), io=24.6MiB (25.7MB), run=11371-11699msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11709: Wed Apr 17 23:33:59 2024 read: IOPS=100, BW=404KiB/s (414kB/s)(5992KiB/14833msec) clat (usec): min=1493, max=149122, avg=4069.32, stdev=4720.80 lat (usec): min=1493, max=149124, avg=4069.99, stdev=4720.84 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1762], 10.00th=[ 1844], 20.00th=[ 1975], | 30.00th=[ 2147], 40.00th=[ 2409], 50.00th=[ 2704], 60.00th=[ 3228], | 70.00th=[ 4080], 80.00th=[ 5669], 90.00th=[ 8356], 95.00th=[ 10421], | 99.00th=[ 14615], 99.50th=[ 15795], 99.90th=[ 18220], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 208, max= 928, per=24.82%, avg=400.83, stdev=174.07, samples=29 iops : min= 52, max= 232, avg=100.21, stdev=43.52, samples=29 write: IOPS=106, BW=424KiB/s (435kB/s)(6296KiB/14833msec) clat (usec): min=2164, max=24064, avg=5523.98, stdev=3231.94 lat (usec): min=2165, max=24065, avg=5524.76, stdev=3231.88 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2606], 10.00th=[ 2802], 20.00th=[ 3032], | 30.00th=[ 3261], 40.00th=[ 3621], 50.00th=[ 4228], 60.00th=[ 5211], | 70.00th=[ 6325], 80.00th=[ 7898], 90.00th=[10028], 95.00th=[11731], | 99.00th=[16909], 99.50th=[20055], 99.90th=[22152], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 232, max= 872, per=24.93%, avg=421.52, stdev=171.41, samples=29 iops : min= 58, max= 218, avg=105.38, stdev=42.85, samples=29 lat (msec) : 2=10.48%, 4=47.49%, 10=33.85%, 20=7.91%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.18%, ctx=3571, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11711: Wed Apr 17 23:33:59 2024 read: IOPS=101, BW=404KiB/s (414kB/s)(5940KiB/14691msec) clat (usec): min=1506, max=148381, avg=4043.65, stdev=4732.76 lat (usec): min=1507, max=148382, avg=4044.32, stdev=4732.74 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1762], 10.00th=[ 1811], 20.00th=[ 1942], | 30.00th=[ 2114], 40.00th=[ 2409], 50.00th=[ 2737], 60.00th=[ 3228], | 70.00th=[ 4015], 80.00th=[ 5473], 90.00th=[ 8160], 95.00th=[ 10421], | 99.00th=[ 15008], 99.50th=[ 16188], 99.90th=[ 19006], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 216, max= 856, per=24.99%, avg=403.66, stdev=176.89, samples=29 iops : min= 54, max= 214, avg=100.79, stdev=44.25, samples=29 write: IOPS=108, BW=432KiB/s (442kB/s)(6348KiB/14691msec) clat (usec): min=2146, max=25594, avg=5446.77, stdev=3199.20 lat (usec): min=2146, max=25595, avg=5447.46, stdev=3199.19 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2638], 10.00th=[ 2769], 20.00th=[ 3032], | 30.00th=[ 3294], 40.00th=[ 3654], 50.00th=[ 4293], 60.00th=[ 5145], | 70.00th=[ 6194], 80.00th=[ 7701], 90.00th=[ 9634], 95.00th=[11600], | 99.00th=[17695], 99.50th=[20055], 99.90th=[25297], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 264, max= 848, per=25.39%, avg=429.31, stdev=171.07, samples=29 iops : min= 66, max= 212, avg=107.21, stdev=42.77, samples=29 lat (msec) : 2=11.46%, 4=46.29%, 10=35.16%, 20=6.84%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.20%, ctx=3528, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11712: Wed Apr 17 23:33:59 2024 read: IOPS=102, BW=409KiB/s (419kB/s)(6028KiB/14726msec) clat (usec): min=1491, max=19277, avg=3924.13, stdev=2808.05 lat (usec): min=1492, max=19278, avg=3924.76, stdev=2808.05 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1745], 10.00th=[ 1827], 20.00th=[ 1958], | 30.00th=[ 2147], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3195], | 70.00th=[ 4146], 80.00th=[ 5407], 90.00th=[ 8225], 95.00th=[10290], | 99.00th=[13960], 99.50th=[15533], 99.90th=[18482], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 176, max= 840, per=25.23%, avg=407.45, stdev=172.64, samples=29 iops : min= 44, max= 210, avg=101.86, stdev=43.16, samples=29 write: IOPS=106, BW=425KiB/s (435kB/s)(6260KiB/14726msec) clat (msec): min=2, max=152, avg= 5.60, stdev= 4.93 lat (msec): min=2, max=152, avg= 5.61, stdev= 4.93 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 26], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 224, max= 816, per=24.99%, avg=422.62, stdev=175.71, samples=29 iops : min= 56, max= 204, avg=105.66, stdev=43.93, samples=29 lat (msec) : 2=11.13%, 4=45.57%, 10=35.58%, 20=7.49%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.14%, ctx=3490, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11713: Wed Apr 17 23:33:59 2024 read: IOPS=101, BW=407KiB/s (417kB/s)(6052KiB/14866msec) clat (usec): min=1516, max=18800, avg=3916.66, stdev=2820.80 lat (usec): min=1517, max=18801, avg=3917.24, stdev=2820.79 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1762], 10.00th=[ 1844], 20.00th=[ 1958], | 30.00th=[ 2147], 40.00th=[ 2376], 50.00th=[ 2737], 60.00th=[ 3163], | 70.00th=[ 4015], 80.00th=[ 5276], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[13960], 99.50th=[15008], 99.90th=[17695], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 208, max= 856, per=24.95%, avg=403.00, stdev=175.90, samples=29 iops : min= 52, max= 214, avg=100.72, stdev=44.00, samples=29 write: IOPS=104, BW=419KiB/s (430kB/s)(6236KiB/14866msec) clat (msec): min=2, max=152, avg= 5.71, stdev= 4.96 lat (msec): min=2, max=152, avg= 5.71, stdev= 4.96 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 26], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 224, max= 856, per=24.57%, avg=415.41, stdev=171.66, samples=29 iops : min= 56, max= 214, avg=103.83, stdev=42.93, samples=29 lat (msec) : 2=10.74%, 4=46.48%, 10=34.64%, 20=7.81%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=11.17%, ctx=3547, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1615KiB/s (1654kB/s), 404KiB/s-409KiB/s (414kB/s-419kB/s), io=23.4MiB (24.6MB), run=14691-14866msec WRITE: bw=1691KiB/s (1732kB/s), 419KiB/s-432KiB/s (430kB/s-442kB/s), io=24.6MiB (25.7MB), run=14691-14866msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11737: Wed Apr 17 23:34:07 2024 read: IOPS=58, BW=932KiB/s (954kB/s)(5920KiB/6355msec) clat (usec): min=62, max=47895, avg=11235.81, stdev=4309.81 lat (usec): min=63, max=47896, avg=11236.43, stdev=4309.83 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 7308], 10.00th=[ 7963], 20.00th=[ 8848], | 30.00th=[ 9503], 40.00th=[10028], 50.00th=[10552], 60.00th=[11469], | 70.00th=[12125], 80.00th=[13173], 90.00th=[15139], 95.00th=[17433], | 99.00th=[27132], 99.50th=[29230], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 672, max= 1088, per=25.27%, avg=930.50, stdev=126.79, samples=12 iops : min= 42, max= 68, avg=58.08, stdev= 7.89, samples=12 write: IOPS=62, BW=1002KiB/s (1026kB/s)(6368KiB/6355msec) clat (usec): min=701, max=101692, avg=5493.44, stdev=7860.81 lat (usec): min=702, max=101693, avg=5494.40, stdev=7860.81 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 791], 10.00th=[ 832], 20.00th=[ 1057], | 30.00th=[ 2057], 40.00th=[ 2442], 50.00th=[ 2900], 60.00th=[ 3884], | 70.00th=[ 5407], 80.00th=[ 7570], 90.00th=[ 13304], 95.00th=[ 16057], | 99.00th=[ 39060], 99.50th=[ 46400], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 672, max= 1344, per=26.30%, avg=1018.50, stdev=238.37, samples=12 iops : min= 42, max= 84, avg=63.58, stdev=14.96, samples=12 lat (usec) : 100=0.52%, 250=0.91%, 750=0.26%, 1000=9.24% lat (msec) : 2=5.08%, 4=16.54%, 10=29.82%, 20=34.77%, 50=2.73% lat (msec) : 250=0.13% cpu : usr=0.16%, sys=20.22%, ctx=2842, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11739: Wed Apr 17 23:34:07 2024 read: IOPS=56, BW=898KiB/s (920kB/s)(5744KiB/6394msec) clat (usec): min=51, max=31543, avg=11047.43, stdev=3835.49 lat (usec): min=51, max=31544, avg=11048.06, stdev=3835.53 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 6915], 10.00th=[ 7701], 20.00th=[ 8586], | 30.00th=[ 9241], 40.00th=[10028], 50.00th=[10552], 60.00th=[11338], | 70.00th=[11994], 80.00th=[13042], 90.00th=[14877], 95.00th=[18220], | 99.00th=[26346], 99.50th=[30278], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 544, max= 1149, per=24.03%, avg=884.67, stdev=193.53, samples=12 iops : min= 34, max= 71, avg=55.08, stdev=11.96, samples=12 write: IOPS=63, BW=1023KiB/s (1048kB/s)(6544KiB/6394msec) clat (usec): min=756, max=136185, avg=5882.43, stdev=10027.42 lat (usec): min=757, max=136186, avg=5883.42, stdev=10027.42 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 1090], | 30.00th=[ 2147], 40.00th=[ 2573], 50.00th=[ 3490], 60.00th=[ 4424], | 70.00th=[ 5800], 80.00th=[ 7701], 90.00th=[ 12780], 95.00th=[ 15533], | 99.00th=[ 34866], 99.50th=[ 56886], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 544, max= 1469, per=25.60%, avg=991.42, stdev=274.33, samples=12 iops : min= 34, max= 91, avg=61.75, stdev=17.12, samples=12 lat (usec) : 100=0.78%, 250=0.13%, 1000=10.03% lat (msec) : 2=4.17%, 4=15.62%, 10=33.72%, 20=32.68%, 50=2.34% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.17%, sys=19.43%, ctx=2855, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11740: Wed Apr 17 23:34:07 2024 read: IOPS=60, BW=962KiB/s (985kB/s)(6256KiB/6505msec) clat (usec): min=58, max=35235, avg=10710.02, stdev=4105.56 lat (usec): min=59, max=35235, avg=10710.66, stdev=4105.64 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 6915], 10.00th=[ 7898], 20.00th=[ 8455], | 30.00th=[ 8979], 40.00th=[ 9634], 50.00th=[10159], 60.00th=[10945], | 70.00th=[11469], 80.00th=[12780], 90.00th=[14615], 95.00th=[16909], | 99.00th=[26084], 99.50th=[32900], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 640, max= 1277, per=26.13%, avg=961.92, stdev=186.98, samples=13 iops : min= 40, max= 79, avg=59.92, stdev=11.64, samples=13 write: IOPS=57, BW=927KiB/s (950kB/s)(6032KiB/6505msec) clat (usec): min=773, max=104554, avg=6087.86, stdev=9711.18 lat (usec): min=774, max=104555, avg=6088.81, stdev=9711.20 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 807], 10.00th=[ 857], 20.00th=[ 1319], | 30.00th=[ 2180], 40.00th=[ 2540], 50.00th=[ 3130], 60.00th=[ 4228], | 70.00th=[ 5473], 80.00th=[ 7570], 90.00th=[ 13435], 95.00th=[ 20317], | 99.00th=[ 52167], 99.50th=[ 84411], 99.90th=[104334], 99.95th=[104334], | 99.99th=[104334] bw ( KiB/s): min= 608, max= 1280, per=23.76%, avg=920.08, stdev=226.22, samples=13 iops : min= 38, max= 80, avg=57.31, stdev=14.13, samples=13 lat (usec) : 100=1.56%, 250=0.26%, 500=0.13%, 1000=7.29% lat (msec) : 2=5.08%, 4=15.49%, 10=36.07%, 20=30.21%, 50=3.39% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.08%, sys=19.16%, ctx=2884, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11741: Wed Apr 17 23:34:07 2024 read: IOPS=58, BW=944KiB/s (967kB/s)(6032KiB/6390msec) clat (usec): min=63, max=87161, avg=11598.33, stdev=6058.37 lat (usec): min=63, max=87162, avg=11599.58, stdev=6058.24 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 7046], 10.00th=[ 7832], 20.00th=[ 8717], | 30.00th=[ 9634], 40.00th=[10159], 50.00th=[10814], 60.00th=[11469], | 70.00th=[12125], 80.00th=[13042], 90.00th=[15008], 95.00th=[17957], | 99.00th=[32113], 99.50th=[52167], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 608, max= 1117, per=25.11%, avg=924.58, stdev=128.96, samples=12 iops : min= 38, max= 69, avg=57.58, stdev= 7.84, samples=12 write: IOPS=61, BW=979KiB/s (1003kB/s)(6256KiB/6390msec) clat (usec): min=762, max=86379, avg=5099.04, stdev=6532.53 lat (usec): min=763, max=86380, avg=5101.21, stdev=6532.54 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 807], 10.00th=[ 857], 20.00th=[ 1696], | 30.00th=[ 2245], 40.00th=[ 2573], 50.00th=[ 3032], 60.00th=[ 3982], | 70.00th=[ 5145], 80.00th=[ 7046], 90.00th=[11863], 95.00th=[14746], | 99.00th=[31589], 99.50th=[43254], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 704, max= 1376, per=25.18%, avg=975.33, stdev=223.05, samples=12 iops : min= 44, max= 86, avg=60.75, stdev=13.94, samples=12 lat (usec) : 100=0.52%, 250=0.26%, 1000=8.20% lat (msec) : 2=3.26%, 4=19.40%, 10=30.86%, 20=34.64%, 50=2.47% lat (msec) : 100=0.39% cpu : usr=0.08%, sys=22.04%, ctx=2964, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3682KiB/s (3770kB/s), 898KiB/s-962KiB/s (920kB/s-985kB/s), io=23.4MiB (24.5MB), run=6355-6505msec WRITE: bw=3874KiB/s (3967kB/s), 927KiB/s-1023KiB/s (950kB/s-1048kB/s), io=24.6MiB (25.8MB), run=6355-6505msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11742: Wed Apr 17 23:34:07 2024 read: IOPS=58, BW=932KiB/s (954kB/s)(5920KiB/6354msec) clat (usec): min=1784, max=54506, avg=6382.29, stdev=5858.52 lat (usec): min=1784, max=54506, avg=6382.94, stdev=5858.63 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2089], 10.00th=[ 2278], 20.00th=[ 2507], | 30.00th=[ 2769], 40.00th=[ 3261], 50.00th=[ 4015], 60.00th=[ 5342], | 70.00th=[ 7767], 80.00th=[ 9372], 90.00th=[12387], 95.00th=[16909], | 99.00th=[26870], 99.50th=[41157], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 736, max= 1088, per=25.31%, avg=930.50, stdev=112.90, samples=12 iops : min= 46, max= 68, avg=58.08, stdev= 7.06, samples=12 write: IOPS=62, BW=1002KiB/s (1026kB/s)(6368KiB/6354msec) clat (msec): min=2, max=108, avg=10.02, stdev= 8.77 lat (msec): min=2, max=108, avg=10.02, stdev= 8.77 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 23], | 99.00th=[ 52], 99.50th=[ 65], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 672, max= 1344, per=26.26%, avg=1015.83, stdev=232.83, samples=12 iops : min= 42, max= 84, avg=63.42, stdev=14.56, samples=12 lat (msec) : 2=1.17%, 4=29.95%, 10=42.45%, 20=21.35%, 50=4.43% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.09%, sys=9.22%, ctx=1201, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11743: Wed Apr 17 23:34:07 2024 read: IOPS=55, BW=888KiB/s (909kB/s)(5744KiB/6468msec) clat (usec): min=1877, max=60362, avg=6769.23, stdev=6244.82 lat (usec): min=1878, max=60362, avg=6769.83, stdev=6244.82 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2442], | 30.00th=[ 2868], 40.00th=[ 3490], 50.00th=[ 4228], 60.00th=[ 6325], | 70.00th=[ 8225], 80.00th=[10552], 90.00th=[13304], 95.00th=[17171], | 99.00th=[29754], 99.50th=[39060], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 510, max= 1088, per=23.77%, avg=873.92, stdev=199.34, samples=12 iops : min= 31, max= 68, avg=54.33, stdev=12.53, samples=12 write: IOPS=63, BW=1012KiB/s (1036kB/s)(6544KiB/6468msec) clat (msec): min=2, max=129, avg= 9.86, stdev=10.13 lat (msec): min=2, max=129, avg= 9.86, stdev=10.13 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 17], 95.00th=[ 21], | 99.00th=[ 40], 99.50th=[ 65], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 510, max= 1472, per=25.28%, avg=977.83, stdev=250.25, samples=12 iops : min= 31, max= 92, avg=60.83, stdev=15.70, samples=12 lat (msec) : 2=1.43%, 4=29.69%, 10=41.02%, 20=23.44%, 50=3.78% lat (msec) : 100=0.39%, 250=0.26% cpu : usr=0.11%, sys=8.81%, ctx=1155, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11744: Wed Apr 17 23:34:07 2024 read: IOPS=60, BW=960KiB/s (983kB/s)(6256KiB/6514msec) clat (usec): min=1837, max=61238, avg=6954.31, stdev=6587.71 lat (usec): min=1837, max=61238, avg=6954.91, stdev=6587.72 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 4359], 60.00th=[ 6128], | 70.00th=[ 8455], 80.00th=[10421], 90.00th=[14615], 95.00th=[19006], | 99.00th=[31851], 99.50th=[42730], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 640, max= 1408, per=25.74%, avg=946.50, stdev=219.75, samples=12 iops : min= 40, max= 88, avg=59.08, stdev=13.72, samples=12 write: IOPS=57, BW=926KiB/s (948kB/s)(6032KiB/6514msec) clat (msec): min=2, max=135, avg=10.05, stdev= 9.42 lat (msec): min=2, max=135, avg=10.05, stdev= 9.42 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 18], 95.00th=[ 21], | 99.00th=[ 39], 99.50th=[ 84], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 608, max= 1184, per=23.37%, avg=903.83, stdev=187.13, samples=12 iops : min= 38, max= 74, avg=56.42, stdev=11.69, samples=12 lat (msec) : 2=1.30%, 4=30.86%, 10=37.89%, 20=24.87%, 50=4.69% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.11%, sys=8.91%, ctx=1193, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11745: Wed Apr 17 23:34:07 2024 read: IOPS=58, BW=941KiB/s (964kB/s)(6032KiB/6407msec) clat (usec): min=1775, max=93324, avg=6561.01, stdev=7284.37 lat (usec): min=1776, max=93325, avg=6561.61, stdev=7284.38 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2507], | 30.00th=[ 2802], 40.00th=[ 3163], 50.00th=[ 3785], 60.00th=[ 4948], | 70.00th=[ 7242], 80.00th=[ 9896], 90.00th=[13042], 95.00th=[18744], | 99.00th=[31065], 99.50th=[45876], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 672, max= 1056, per=25.14%, avg=924.25, stdev=128.38, samples=12 iops : min= 42, max= 66, avg=57.42, stdev= 7.87, samples=12 write: IOPS=61, BW=976KiB/s (1000kB/s)(6256KiB/6407msec) clat (usec): min=2676, max=96778, avg=10041.25, stdev=7914.66 lat (usec): min=2677, max=96781, avg=10042.31, stdev=7914.70 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3458], 10.00th=[ 3818], 20.00th=[ 4621], | 30.00th=[ 5932], 40.00th=[ 6980], 50.00th=[ 8225], 60.00th=[ 9372], | 70.00th=[11207], 80.00th=[14091], 90.00th=[17433], 95.00th=[22414], | 99.00th=[41157], 99.50th=[59507], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 640, max= 1341, per=25.20%, avg=974.83, stdev=227.02, samples=12 iops : min= 40, max= 83, avg=60.58, stdev=14.04, samples=12 lat (msec) : 2=1.43%, 4=30.21%, 10=40.62%, 20=22.40%, 50=4.95% lat (msec) : 100=0.39% cpu : usr=0.11%, sys=8.99%, ctx=1205, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3677KiB/s (3765kB/s), 888KiB/s-960KiB/s (909kB/s-983kB/s), io=23.4MiB (24.5MB), run=6354-6514msec WRITE: bw=3869KiB/s (3961kB/s), 926KiB/s-1012KiB/s (948kB/s-1036kB/s), io=24.6MiB (25.8MB), run=6354-6514msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11756: Wed Apr 17 23:34:13 2024 read: IOPS=1, BW=1623KiB/s (1662kB/s)(7168KiB/4416msec) clat (msec): min=26, max=778, avg=473.51, stdev=367.47 lat (msec): min=26, max=778, avg=473.51, stdev=367.47 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 35], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 743], 60.00th=[ 760], | 70.00th=[ 760], 80.00th=[ 776], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 2048, max= 2048, per=51.55%, avg=2048.00, stdev= 0.00, samples=5 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=5 write: IOPS=1, BW=1159KiB/s (1187kB/s)(5120KiB/4416msec) clat (msec): min=31, max=828, avg=216.43, stdev=344.11 lat (msec): min=31, max=828, avg=216.52, stdev=344.10 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 51], 60.00th=[ 51], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 2048, max= 4096, per=49.09%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 50=41.67%, 250=16.67%, 750=8.33%, 1000=33.33% cpu : usr=0.00%, sys=19.00%, ctx=1137, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11757: Wed Apr 17 23:34:13 2024 read: IOPS=0, BW=464KiB/s (475kB/s)(2048KiB/4413msec) clat (msec): min=34, max=819, avg=427.16, stdev=554.62 lat (msec): min=34, max=819, avg=427.16, stdev=554.62 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 818], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 2048, max= 2048, per=51.55%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2320KiB/s (2376kB/s)(10.0MiB/4413msec) clat (msec): min=28, max=869, avg=354.43, stdev=378.95 lat (msec): min=28, max=870, avg=354.49, stdev=378.94 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 79], 60.00th=[ 171], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 810], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 4096, max= 6144, per=82.83%, avg=4608.00, stdev=1024.00, samples=4 iops : min= 4, max= 6, avg= 4.50, stdev= 1.00, samples=4 lat (msec) : 50=41.67%, 100=8.33%, 250=8.33%, 750=8.33%, 1000=33.33% cpu : usr=0.00%, sys=11.49%, ctx=385, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11758: Wed Apr 17 23:34:13 2024 read: IOPS=1, BW=1192KiB/s (1221kB/s)(6144KiB/5153msec) clat (msec): min=16, max=765, avg=281.75, stdev=362.87 lat (msec): min=16, max=765, avg=281.75, stdev=362.87 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 68], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2048, max= 4096, per=85.91%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1192KiB/s (1221kB/s)(6144KiB/5153msec) clat (msec): min=23, max=1601, avg=572.75, stdev=643.86 lat (msec): min=23, max=1601, avg=572.79, stdev=643.84 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 818], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 2048, max= 4087, per=44.15%, avg=2455.80, stdev=911.87, samples=5 iops : min= 2, max= 3, avg= 2.20, stdev= 0.45, samples=5 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 750=8.33%, 1000=25.00% cpu : usr=0.00%, sys=9.39%, ctx=696, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11759: Wed Apr 17 23:34:13 2024 read: IOPS=0, BW=993KiB/s (1017kB/s)(5120KiB/5154msec) clat (msec): min=67, max=758, avg=433.30, stdev=335.86 lat (msec): min=67, max=758, avg=433.30, stdev=335.86 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 100], 40.00th=[ 100], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 2048, max= 4096, per=68.73%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1391KiB/s (1424kB/s)(7168KiB/5154msec) clat (msec): min=66, max=1584, avg=421.91, stdev=609.82 lat (msec): min=66, max=1584, avg=421.97, stdev=609.82 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 82], | 30.00th=[ 84], 40.00th=[ 84], 50.00th=[ 85], 60.00th=[ 87], | 70.00th=[ 87], 80.00th=[ 969], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 2048, max= 4096, per=51.54%, avg=2867.20, stdev=1121.74, samples=5 iops : min= 2, max= 4, avg= 2.80, stdev= 1.10, samples=5 lat (msec) : 100=58.33%, 500=8.33%, 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=8.52%, ctx=546, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3974KiB/s (4069kB/s), 464KiB/s-1623KiB/s (475kB/s-1662kB/s), io=20.0MiB (20.0MB), run=4413-5154msec WRITE: bw=5563KiB/s (5697kB/s), 1159KiB/s-2320KiB/s (1187kB/s-2376kB/s), io=28.0MiB (29.4MB), run=4413-5154msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11760: Wed Apr 17 23:34:13 2024 read: IOPS=1, BW=2016KiB/s (2065kB/s)(7168KiB/3555msec) clat (msec): min=23, max=781, avg=243.63, stdev=357.10 lat (msec): min=23, max=781, avg=243.63, stdev=357.10 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 40], 60.00th=[ 42], | 70.00th=[ 42], 80.00th=[ 751], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 2048, max= 4096, per=78.69%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 write: IOPS=1, BW=1440KiB/s (1475kB/s)(5120KiB/3555msec) clat (msec): min=24, max=969, avg=368.70, stdev=469.25 lat (msec): min=24, max=969, avg=368.75, stdev=469.25 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 2048, max= 4096, per=49.96%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 50=66.67%, 1000=33.33% cpu : usr=0.00%, sys=3.57%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11761: Wed Apr 17 23:34:13 2024 read: IOPS=0, BW=390KiB/s (400kB/s)(2048KiB/5245msec) clat (usec): min=41328, max=89347, avg=65337.87, stdev=33954.67 lat (usec): min=41329, max=89348, avg=65339.04, stdev=33955.18 clat percentiles (usec): | 1.00th=[41157], 5.00th=[41157], 10.00th=[41157], 20.00th=[41157], | 30.00th=[41157], 40.00th=[41157], 50.00th=[41157], 60.00th=[89654], | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 1726, max= 2048, per=48.34%, avg=1887.00, stdev=227.69, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1952KiB/s (1999kB/s)(10.0MiB/5245msec) clat (msec): min=27, max=1591, avg=510.21, stdev=535.53 lat (msec): min=27, max=1591, avg=510.27, stdev=535.53 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 35], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 93], 60.00th=[ 584], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 961], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 1726, max= 4096, per=52.89%, avg=2890.80, stdev=1095.24, samples=5 iops : min= 1, max= 4, avg= 2.40, stdev= 1.34, samples=5 lat (msec) : 50=25.00%, 100=33.33%, 750=8.33%, 1000=25.00% cpu : usr=0.00%, sys=3.53%, ctx=51, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11762: Wed Apr 17 23:34:13 2024 read: IOPS=1, BW=1186KiB/s (1215kB/s)(6144KiB/5179msec) clat (msec): min=20, max=994, avg=541.48, stdev=401.97 lat (msec): min=20, max=994, avg=541.48, stdev=401.97 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 743], | 70.00th=[ 810], 80.00th=[ 810], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 1723, max= 3447, per=57.26%, avg=2235.25, stdev=822.23, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 write: IOPS=1, BW=1186KiB/s (1215kB/s)(6144KiB/5179msec) clat (msec): min=39, max=1543, avg=319.53, stdev=600.89 lat (msec): min=39, max=1543, avg=319.62, stdev=600.86 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 1723, max= 4096, per=51.75%, avg=2828.50, stdev=1128.48, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 lat (msec) : 50=16.67%, 100=33.33%, 250=8.33%, 750=16.67%, 1000=16.67% cpu : usr=0.02%, sys=3.63%, ctx=74, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11763: Wed Apr 17 23:34:13 2024 read: IOPS=1, BW=1785KiB/s (1827kB/s)(5120KiB/2869msec) clat (usec): min=19052, max=50491, avg=26260.36, stdev=13568.36 lat (usec): min=19053, max=50492, avg=26261.35, stdev=13568.15 clat percentiles (usec): | 1.00th=[19006], 5.00th=[19006], 10.00th=[19006], 20.00th=[19006], | 30.00th=[19792], 40.00th=[19792], 50.00th=[20841], 60.00th=[20841], | 70.00th=[21103], 80.00th=[21103], 90.00th=[50594], 95.00th=[50594], | 99.00th=[50594], 99.50th=[50594], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2498KiB/s (2558kB/s)(7168KiB/2869msec) clat (msec): min=42, max=1577, avg=389.33, stdev=591.99 lat (msec): min=42, max=1577, avg=389.39, stdev=591.99 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 63], 60.00th=[ 127], | 70.00th=[ 127], 80.00th=[ 802], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 2048, max= 6144, per=74.94%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.40%, ctx=72, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3905KiB/s (3998kB/s), 390KiB/s-2016KiB/s (400kB/s-2065kB/s), io=20.0MiB (20.0MB), run=2869-5245msec WRITE: bw=5467KiB/s (5598kB/s), 1186KiB/s-2498KiB/s (1215kB/s-2558kB/s), io=28.0MiB (29.4MB), run=2869-5245msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11772: Wed Apr 17 23:34:16 2024 read: IOPS=2, BW=8416KiB/s (8618kB/s)(12.0MiB/1460msec) clat (msec): min=167, max=792, avg=485.30, stdev=312.72 lat (msec): min=167, max=792, avg=485.30, stdev=312.72 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 498], 50.00th=[ 498], 60.00th=[ 498], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 7111, max= 8159, per=87.24%, avg=7635.00, stdev=741.05, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=83.07%, ctx=204, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11773: Wed Apr 17 23:34:16 2024 read: IOPS=0, BW=2188KiB/s (2241kB/s)(4096KiB/1872msec) clat (nsec): min=94629k, max=94629k, avg=94629198.00, stdev= 0.00 lat (nsec): min=94632k, max=94632k, avg=94631503.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[94897], 5.00th=[94897], 10.00th=[94897], 20.00th=[94897], | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897], | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 8159, max= 8159, per=93.22%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4376KiB/s (4481kB/s)(8192KiB/1872msec) clat (msec): min=405, max=1351, avg=878.33, stdev=668.47 lat (msec): min=405, max=1351, avg=878.52, stdev=668.39 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7211, max= 7211, per=41.20%, avg=7211.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.85%, ctx=61, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11774: Wed Apr 17 23:34:16 2024 write: IOPS=1, BW=7111KiB/s (7282kB/s)(12.0MiB/1728msec) clat (msec): min=228, max=939, avg=574.21, stdev=355.76 lat (msec): min=228, max=939, avg=574.40, stdev=355.75 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 936], 80.00th=[ 936], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7074, max= 8126, per=43.42%, avg=7600.00, stdev=743.88, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=14.18%, ctx=47, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11775: Wed Apr 17 23:34:16 2024 write: IOPS=5, BW=20.0MiB/s (21.0MB/s)(12.0MiB/599msec) clat (msec): min=128, max=256, avg=192.20, stdev=64.11 lat (msec): min=128, max=257, avg=192.62, stdev=64.34 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min=16384, max=16384, per=93.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=46.32%, ctx=66, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8752KiB/s (8962kB/s), 2188KiB/s-8416KiB/s (2241kB/s-8618kB/s), io=16.0MiB (16.8MB), run=1460-1872msec WRITE: bw=17.1MiB/s (17.9MB/s), 4376KiB/s-20.0MiB/s (4481kB/s-21.0MB/s), io=32.0MiB (33.6MB), run=599-1872msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11777: Wed Apr 17 23:34:16 2024 read: IOPS=1, BW=6919KiB/s (7085kB/s)(12.0MiB/1776msec) clat (msec): min=87, max=1290, avg=591.07, stdev=624.99 lat (msec): min=87, max=1290, avg=591.07, stdev=624.99 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8192, max= 8192, per=95.46%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.06%, sys=4.56%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11778: Wed Apr 17 23:34:16 2024 read: IOPS=0, BW=2146KiB/s (2197kB/s)(4096KiB/1909msec) clat (nsec): min=359718k, max=359718k, avg=359717884.00, stdev= 0.00 lat (nsec): min=359719k, max=359719k, avg=359718813.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8159, max= 8159, per=95.07%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4291KiB/s (4394kB/s)(8192KiB/1909msec) clat (msec): min=198, max=1343, avg=771.47, stdev=809.63 lat (msec): min=199, max=1344, avg=771.73, stdev=809.65 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8192, max= 8192, per=47.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.05%, sys=7.86%, ctx=110, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11779: Wed Apr 17 23:34:16 2024 write: IOPS=1, BW=6689KiB/s (6850kB/s)(12.0MiB/1837msec) clat (msec): min=218, max=1268, avg=603.54, stdev=577.96 lat (msec): min=218, max=1268, avg=603.81, stdev=577.96 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8175, max= 8175, per=47.63%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=9.15%, ctx=59, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11780: Wed Apr 17 23:34:16 2024 write: IOPS=1, BW=7103KiB/s (7273kB/s)(12.0MiB/1730msec) clat (msec): min=87, max=1304, avg=572.70, stdev=644.96 lat (msec): min=87, max=1304, avg=572.98, stdev=644.96 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=47.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=8.04%, ctx=90, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8583KiB/s (8788kB/s), 2146KiB/s-6919KiB/s (2197kB/s-7085kB/s), io=16.0MiB (16.8MB), run=1776-1909msec WRITE: bw=16.8MiB/s (17.6MB/s), 4291KiB/s-7103KiB/s (4394kB/s-7273kB/s), io=32.0MiB (33.6MB), run=1730-1909msec