== sanity test 398b: DIO and buffer IO race ============== 17:32:46 (1713303166) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.3441 s, 37.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12157: Tue Apr 16 17:33:03 2024 read: IOPS=103, BW=414KiB/s (424kB/s)(5992KiB/14475msec) clat (usec): min=56, max=194946, avg=5532.82, stdev=5706.14 lat (usec): min=57, max=194947, avg=5533.52, stdev=5706.14 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 2245], 10.00th=[ 2606], 20.00th=[ 3195], | 30.00th=[ 3720], 40.00th=[ 4293], 50.00th=[ 4948], 60.00th=[ 5735], | 70.00th=[ 6456], 80.00th=[ 7308], 90.00th=[ 8848], 95.00th=[ 10421], | 99.00th=[ 15664], 99.50th=[ 17695], 99.90th=[ 27132], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 232, max= 512, per=24.39%, avg=404.46, stdev=56.29, samples=28 iops : min= 58, max= 128, avg=101.07, stdev=14.04, samples=28 write: IOPS=108, BW=435KiB/s (445kB/s)(6296KiB/14475msec) clat (usec): min=432, max=24723, avg=3890.41, stdev=3417.03 lat (usec): min=433, max=24725, avg=3891.25, stdev=3417.08 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 594], 10.00th=[ 660], 20.00th=[ 807], | 30.00th=[ 1029], 40.00th=[ 2278], 50.00th=[ 3195], 60.00th=[ 3884], | 70.00th=[ 4817], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[10552], | 99.00th=[13698], 99.50th=[16188], 99.90th=[21365], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 296, max= 560, per=24.42%, avg=423.89, stdev=56.44, samples=28 iops : min= 74, max= 140, avg=105.93, stdev=14.12, samples=28 lat (usec) : 100=1.30%, 250=0.68%, 500=0.13%, 750=8.53%, 1000=6.41% lat (msec) : 2=4.98%, 4=26.60%, 10=45.51%, 20=5.66%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=23.86%, ctx=6260, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12158: Tue Apr 16 17:33:03 2024 read: IOPS=103, BW=414KiB/s (424kB/s)(5940KiB/14356msec) clat (usec): min=50, max=22835, avg=5349.92, stdev=2795.07 lat (usec): min=50, max=22836, avg=5351.14, stdev=2794.81 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 2311], 10.00th=[ 2737], 20.00th=[ 3195], | 30.00th=[ 3621], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5538], | 70.00th=[ 6390], 80.00th=[ 7439], 90.00th=[ 8717], 95.00th=[10421], | 99.00th=[14615], 99.50th=[15795], 99.90th=[22676], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 256, max= 528, per=24.88%, avg=412.46, stdev=57.26, samples=28 iops : min= 64, max= 132, avg=103.07, stdev=14.32, samples=28 write: IOPS=110, BW=442KiB/s (453kB/s)(6348KiB/14356msec) clat (usec): min=478, max=194079, avg=3995.29, stdev=5779.85 lat (usec): min=479, max=194080, avg=3997.90, stdev=5779.61 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 603], 10.00th=[ 668], 20.00th=[ 816], | 30.00th=[ 1057], 40.00th=[ 2540], 50.00th=[ 3228], 60.00th=[ 3916], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 8848], 95.00th=[ 10290], | 99.00th=[ 13960], 99.50th=[ 15139], 99.90th=[ 20841], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 304, max= 576, per=25.11%, avg=435.89, stdev=68.61, samples=28 iops : min= 76, max= 144, avg=108.93, stdev=17.16, samples=28 lat (usec) : 100=0.98%, 250=0.85%, 500=0.16%, 750=8.11%, 1000=6.77% lat (msec) : 2=3.91%, 4=28.65%, 10=45.02%, 20=5.40%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=24.28%, ctx=6302, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12159: Tue Apr 16 17:33:03 2024 read: IOPS=104, BW=417KiB/s (427kB/s)(6028KiB/14463msec) clat (usec): min=50, max=35799, avg=5220.51, stdev=2904.32 lat (usec): min=50, max=35800, avg=5222.06, stdev=2904.45 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 465], 10.00th=[ 2638], 20.00th=[ 3130], | 30.00th=[ 3621], 40.00th=[ 4146], 50.00th=[ 4752], 60.00th=[ 5407], | 70.00th=[ 6259], 80.00th=[ 7111], 90.00th=[ 8356], 95.00th=[ 9896], | 99.00th=[15664], 99.50th=[17171], 99.90th=[25297], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 264, max= 504, per=24.67%, avg=409.04, stdev=50.63, samples=28 iops : min= 66, max= 126, avg=102.21, stdev=12.69, samples=28 write: IOPS=108, BW=433KiB/s (443kB/s)(6260KiB/14463msec) clat (usec): min=477, max=170546, avg=4170.72, stdev=5495.75 lat (usec): min=477, max=170548, avg=4172.23, stdev=5495.98 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 611], 10.00th=[ 660], 20.00th=[ 824], | 30.00th=[ 1057], 40.00th=[ 2638], 50.00th=[ 3326], 60.00th=[ 4015], | 70.00th=[ 5211], 80.00th=[ 6718], 90.00th=[ 8979], 95.00th=[ 10945], | 99.00th=[ 15139], 99.50th=[ 17695], 99.90th=[ 25297], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 328, max= 528, per=24.37%, avg=423.04, stdev=61.80, samples=28 iops : min= 82, max= 132, avg=105.71, stdev=15.42, samples=28 lat (usec) : 100=1.60%, 250=0.75%, 500=0.23%, 750=8.24%, 1000=6.22% lat (msec) : 2=4.07%, 4=27.60%, 10=45.54%, 20=5.53%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.76%, ctx=6332, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12160: Tue Apr 16 17:33:03 2024 read: IOPS=105, BW=422KiB/s (432kB/s)(6052KiB/14350msec) clat (usec): min=53, max=24249, avg=5292.58, stdev=2873.97 lat (usec): min=53, max=24250, avg=5294.81, stdev=2873.66 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 2212], 10.00th=[ 2606], 20.00th=[ 3163], | 30.00th=[ 3654], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 5538], | 70.00th=[ 6259], 80.00th=[ 7046], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[16319], 99.50th=[19268], 99.90th=[21627], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 248, max= 544, per=25.12%, avg=416.46, stdev=59.95, samples=28 iops : min= 62, max= 136, avg=104.07, stdev=14.99, samples=28 write: IOPS=108, BW=435KiB/s (445kB/s)(6236KiB/14350msec) clat (usec): min=443, max=160963, avg=4023.92, stdev=5297.51 lat (usec): min=444, max=160964, avg=4026.68, stdev=5296.55 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 611], 10.00th=[ 660], 20.00th=[ 783], | 30.00th=[ 971], 40.00th=[ 2212], 50.00th=[ 3294], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6783], 90.00th=[ 8586], 95.00th=[ 10683], | 99.00th=[ 14746], 99.50th=[ 17433], 99.90th=[ 31589], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 320, max= 560, per=24.73%, avg=429.32, stdev=54.82, samples=28 iops : min= 80, max= 140, avg=107.29, stdev=13.69, samples=28 lat (usec) : 100=1.24%, 250=0.78%, 500=0.16%, 750=8.89%, 1000=6.93% lat (msec) : 2=4.10%, 4=26.82%, 10=45.21%, 20=5.60%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.79%, ctx=6172, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1659KiB/s (1699kB/s), 414KiB/s-422KiB/s (424kB/s-432kB/s), io=23.4MiB (24.6MB), run=14350-14475msec WRITE: bw=1737KiB/s (1778kB/s), 433KiB/s-442KiB/s (443kB/s-453kB/s), io=24.6MiB (25.7MB), run=14350-14475msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12153: Tue Apr 16 17:33:07 2024 read: IOPS=84, BW=337KiB/s (345kB/s)(5992KiB/17767msec) clat (usec): min=1571, max=197400, avg=4832.76, stdev=6163.63 lat (usec): min=1571, max=197401, avg=4833.49, stdev=6163.64 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 6521], 90.00th=[ 10028], 95.00th=[ 12649], | 99.00th=[ 19268], 99.50th=[ 21890], 99.90th=[ 27132], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 152, max= 824, per=24.78%, avg=333.54, stdev=165.58, samples=35 iops : min= 38, max= 206, avg=83.29, stdev=41.42, samples=35 write: IOPS=88, BW=354KiB/s (363kB/s)(6296KiB/17767msec) clat (usec): min=2128, max=28706, avg=6650.33, stdev=4150.52 lat (usec): min=2129, max=28707, avg=6651.32, stdev=4150.49 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3425], | 30.00th=[ 3752], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 6259], | 70.00th=[ 7963], 80.00th=[ 9765], 90.00th=[12387], 95.00th=[14746], | 99.00th=[22414], 99.50th=[24511], 99.90th=[27395], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 168, max= 798, per=24.86%, avg=350.23, stdev=170.27, samples=35 iops : min= 42, max= 199, avg=87.46, stdev=42.56, samples=35 lat (msec) : 2=3.32%, 4=45.35%, 10=36.72%, 20=13.57%, 50=1.01% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.48%, ctx=3503, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12154: Tue Apr 16 17:33:07 2024 read: IOPS=83, BW=333KiB/s (341kB/s)(5940KiB/17838msec) clat (usec): min=1636, max=164697, avg=4861.97, stdev=5570.10 lat (usec): min=1636, max=164698, avg=4862.67, stdev=5570.12 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3851], | 70.00th=[ 4752], 80.00th=[ 6783], 90.00th=[ 10552], 95.00th=[ 12649], | 99.00th=[ 18220], 99.50th=[ 22414], 99.90th=[ 33162], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 104, max= 728, per=24.56%, avg=330.63, stdev=170.66, samples=35 iops : min= 26, max= 182, avg=82.57, stdev=42.72, samples=35 write: IOPS=88, BW=356KiB/s (364kB/s)(6348KiB/17838msec) clat (usec): min=2264, max=41593, avg=6653.47, stdev=4134.90 lat (usec): min=2265, max=41595, avg=6654.34, stdev=4134.91 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3458], | 30.00th=[ 3720], 40.00th=[ 4293], 50.00th=[ 5080], 60.00th=[ 6325], | 70.00th=[ 7832], 80.00th=[ 9634], 90.00th=[12387], 95.00th=[14353], | 99.00th=[21627], 99.50th=[23987], 99.90th=[29492], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 168, max= 768, per=24.75%, avg=348.69, stdev=159.60, samples=35 iops : min= 42, max= 192, avg=87.09, stdev=39.93, samples=35 lat (msec) : 2=3.16%, 4=45.25%, 10=36.91%, 20=13.67%, 50=0.98% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.38%, ctx=3507, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12155: Tue Apr 16 17:33:07 2024 read: IOPS=85, BW=340KiB/s (348kB/s)(6028KiB/17728msec) clat (usec): min=1670, max=23378, avg=4765.19, stdev=3642.48 lat (usec): min=1671, max=23379, avg=4765.86, stdev=3642.49 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3884], | 70.00th=[ 5014], 80.00th=[ 6915], 90.00th=[10028], 95.00th=[12518], | 99.00th=[18482], 99.50th=[20317], 99.90th=[22676], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 128, max= 744, per=25.08%, avg=337.54, stdev=168.18, samples=35 iops : min= 32, max= 186, avg=84.34, stdev=42.06, samples=35 write: IOPS=88, BW=353KiB/s (362kB/s)(6260KiB/17728msec) clat (msec): min=2, max=170, avg= 6.70, stdev= 5.85 lat (msec): min=2, max=170, avg= 6.70, stdev= 5.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 26], 99.90th=[ 39], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 176, max= 728, per=24.80%, avg=349.43, stdev=168.22, samples=35 iops : min= 44, max= 182, avg=87.31, stdev=42.07, samples=35 lat (msec) : 2=3.91%, 4=44.76%, 10=36.85%, 20=13.67%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=11.59%, ctx=3492, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12156: Tue Apr 16 17:33:07 2024 read: IOPS=85, BW=341KiB/s (349kB/s)(6052KiB/17750msec) clat (usec): min=1569, max=24250, avg=4677.16, stdev=3517.51 lat (usec): min=1569, max=24251, avg=4677.96, stdev=3517.59 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 6783], 90.00th=[10290], 95.00th=[12256], | 99.00th=[16909], 99.50th=[19792], 99.90th=[23725], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 152, max= 776, per=25.10%, avg=337.89, stdev=165.11, samples=35 iops : min= 38, max= 194, avg=84.37, stdev=41.28, samples=35 write: IOPS=87, BW=351KiB/s (360kB/s)(6236KiB/17750msec) clat (msec): min=2, max=165, avg= 6.81, stdev= 5.89 lat (msec): min=2, max=165, avg= 6.81, stdev= 5.89 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 22], 99.50th=[ 27], 99.90th=[ 39], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 184, max= 784, per=24.69%, avg=347.94, stdev=167.26, samples=35 iops : min= 46, max= 196, avg=86.89, stdev=41.82, samples=35 lat (msec) : 2=3.22%, 4=45.77%, 10=36.33%, 20=13.67%, 50=0.98% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.57%, ctx=3523, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1346KiB/s (1378kB/s), 333KiB/s-341KiB/s (341kB/s-349kB/s), io=23.4MiB (24.6MB), run=17728-17838msec WRITE: bw=1409KiB/s (1443kB/s), 351KiB/s-356KiB/s (360kB/s-364kB/s), io=24.6MiB (25.7MB), run=17728-17838msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12187: Tue Apr 16 17:33:14 2024 read: IOPS=62, BW=994KiB/s (1018kB/s)(5920KiB/5956msec) clat (usec): min=2097, max=27573, avg=6667.88, stdev=4539.34 lat (usec): min=2098, max=27574, avg=6668.56, stdev=4539.34 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2606], 10.00th=[ 2769], 20.00th=[ 3064], | 30.00th=[ 3392], 40.00th=[ 3916], 50.00th=[ 4752], 60.00th=[ 6259], | 70.00th=[ 8094], 80.00th=[10552], 90.00th=[12649], 95.00th=[15008], | 99.00th=[21365], 99.50th=[25560], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 672, max= 1312, per=24.16%, avg=971.64, stdev=178.33, samples=11 iops : min= 42, max= 82, avg=60.73, stdev=11.15, samples=11 write: IOPS=66, BW=1069KiB/s (1095kB/s)(6368KiB/5956msec) clat (usec): min=3062, max=28999, avg=8743.15, stdev=5265.57 lat (usec): min=3092, max=29000, avg=8744.63, stdev=5265.61 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 3785], 10.00th=[ 4047], 20.00th=[ 4490], | 30.00th=[ 4883], 40.00th=[ 5735], 50.00th=[ 6652], 60.00th=[ 8029], | 70.00th=[10552], 80.00th=[13304], 90.00th=[16909], 95.00th=[19530], | 99.00th=[26084], 99.50th=[28443], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 800, max= 1344, per=25.03%, avg=1058.91, stdev=163.45, samples=11 iops : min= 50, max= 84, avg=66.18, stdev=10.22, samples=11 lat (msec) : 4=25.13%, 10=46.35%, 20=25.39%, 50=3.12% cpu : usr=0.35%, sys=10.01%, ctx=919, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12188: Tue Apr 16 17:33:14 2024 read: IOPS=61, BW=978KiB/s (1001kB/s)(5744KiB/5876msec) clat (usec): min=2170, max=26936, avg=6399.43, stdev=4144.12 lat (usec): min=2171, max=26937, avg=6400.28, stdev=4144.01 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2540], 10.00th=[ 2769], 20.00th=[ 3163], | 30.00th=[ 3458], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5735], | 70.00th=[ 7504], 80.00th=[10028], 90.00th=[12125], 95.00th=[14222], | 99.00th=[18220], 99.50th=[24249], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 672, max= 1216, per=24.23%, avg=974.09, stdev=164.63, samples=11 iops : min= 42, max= 76, avg=60.73, stdev=10.32, samples=11 write: IOPS=69, BW=1114KiB/s (1140kB/s)(6544KiB/5876msec) clat (usec): min=3143, max=33355, avg=8732.29, stdev=5320.68 lat (usec): min=3145, max=33357, avg=8733.83, stdev=5320.93 clat percentiles (usec): | 1.00th=[ 3359], 5.00th=[ 3720], 10.00th=[ 4047], 20.00th=[ 4490], | 30.00th=[ 5145], 40.00th=[ 5932], 50.00th=[ 6980], 60.00th=[ 7898], | 70.00th=[10028], 80.00th=[12256], 90.00th=[16450], 95.00th=[19792], | 99.00th=[25560], 99.50th=[28705], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 960, max= 1280, per=26.05%, avg=1102.00, stdev=103.65, samples=11 iops : min= 60, max= 80, avg=68.73, stdev= 6.51, samples=11 lat (msec) : 4=24.61%, 10=49.74%, 20=22.79%, 50=2.86% cpu : usr=0.12%, sys=10.40%, ctx=897, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12189: Tue Apr 16 17:33:14 2024 read: IOPS=67, BW=1081KiB/s (1107kB/s)(6256KiB/5786msec) clat (usec): min=2105, max=30885, avg=6497.45, stdev=4494.71 lat (usec): min=2106, max=30885, avg=6498.18, stdev=4494.71 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2474], 10.00th=[ 2671], 20.00th=[ 3032], | 30.00th=[ 3359], 40.00th=[ 3916], 50.00th=[ 4490], 60.00th=[ 5538], | 70.00th=[ 7767], 80.00th=[10290], 90.00th=[12780], 95.00th=[15008], | 99.00th=[21365], 99.50th=[23462], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 832, max= 1472, per=26.99%, avg=1085.09, stdev=196.98, samples=11 iops : min= 52, max= 92, avg=67.82, stdev=12.31, samples=11 write: IOPS=65, BW=1043KiB/s (1068kB/s)(6032KiB/5786msec) clat (usec): min=2951, max=35503, avg=8585.40, stdev=5181.65 lat (usec): min=2953, max=35504, avg=8586.88, stdev=5181.60 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3523], 10.00th=[ 3982], 20.00th=[ 4490], | 30.00th=[ 4883], 40.00th=[ 5735], 50.00th=[ 6849], 60.00th=[ 8225], | 70.00th=[10028], 80.00th=[11863], 90.00th=[16057], 95.00th=[19268], | 99.00th=[25560], 99.50th=[28443], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 800, max= 1248, per=24.06%, avg=1018.18, stdev=130.24, samples=11 iops : min= 50, max= 78, avg=63.64, stdev= 8.14, samples=11 lat (msec) : 4=27.21%, 10=46.74%, 20=22.92%, 50=3.12% cpu : usr=0.21%, sys=10.34%, ctx=901, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12190: Tue Apr 16 17:33:14 2024 read: IOPS=63, BW=1015KiB/s (1039kB/s)(6032KiB/5945msec) clat (usec): min=2148, max=26039, avg=6761.43, stdev=4811.03 lat (usec): min=2149, max=26039, avg=6762.10, stdev=4811.04 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2540], 10.00th=[ 2704], 20.00th=[ 3032], | 30.00th=[ 3392], 40.00th=[ 3785], 50.00th=[ 4490], 60.00th=[ 5932], | 70.00th=[ 8291], 80.00th=[10683], 90.00th=[13698], 95.00th=[16319], | 99.00th=[24511], 99.50th=[25560], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 704, max= 1184, per=24.82%, avg=997.82, stdev=138.61, samples=11 iops : min= 44, max= 74, avg=62.36, stdev= 8.66, samples=11 write: IOPS=65, BW=1052KiB/s (1078kB/s)(6256KiB/5945msec) clat (usec): min=3253, max=29157, avg=8665.37, stdev=5310.54 lat (usec): min=3253, max=29158, avg=8666.99, stdev=5310.62 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 3884], 10.00th=[ 4113], 20.00th=[ 4424], | 30.00th=[ 4883], 40.00th=[ 5407], 50.00th=[ 6521], 60.00th=[ 7832], | 70.00th=[10028], 80.00th=[12518], 90.00th=[17171], 95.00th=[19006], | 99.00th=[25822], 99.50th=[28181], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 832, max= 1248, per=24.89%, avg=1053.09, stdev=116.66, samples=11 iops : min= 52, max= 78, avg=65.82, stdev= 7.29, samples=11 lat (msec) : 4=25.65%, 10=47.27%, 20=24.09%, 50=2.99% cpu : usr=0.22%, sys=9.94%, ctx=920, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4021KiB/s (4118kB/s), 978KiB/s-1081KiB/s (1001kB/s-1107kB/s), io=23.4MiB (24.5MB), run=5786-5956msec WRITE: bw=4231KiB/s (4333kB/s), 1043KiB/s-1114KiB/s (1068kB/s-1140kB/s), io=24.6MiB (25.8MB), run=5786-5956msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12191: Tue Apr 16 17:33:14 2024 read: IOPS=58, BW=931KiB/s (954kB/s)(5920KiB/6356msec) clat (usec): min=57, max=31476, avg=12438.33, stdev=4988.60 lat (usec): min=57, max=31476, avg=12439.18, stdev=4988.68 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 108], 10.00th=[ 7832], 20.00th=[10028], | 30.00th=[11076], 40.00th=[11863], 50.00th=[12780], 60.00th=[13829], | 70.00th=[14746], 80.00th=[15795], 90.00th=[17433], 95.00th=[19268], | 99.00th=[23987], 99.50th=[26608], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 736, max= 1149, per=23.36%, avg=876.75, stdev=123.34, samples=12 iops : min= 46, max= 71, avg=54.58, stdev= 7.57, samples=12 write: IOPS=62, BW=1002KiB/s (1026kB/s)(6368KiB/6356msec) clat (usec): min=750, max=29807, avg=4379.23, stdev=4232.12 lat (usec): min=750, max=29808, avg=4380.50, stdev=4232.07 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 873], 10.00th=[ 971], 20.00th=[ 1205], | 30.00th=[ 2089], 40.00th=[ 2802], 50.00th=[ 3261], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 5669], 90.00th=[10028], 95.00th=[13566], | 99.00th=[21103], 99.50th=[27132], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 510, max= 1469, per=24.37%, avg=962.08, stdev=257.83, samples=12 iops : min= 31, max= 91, avg=59.92, stdev=16.14, samples=12 lat (usec) : 100=2.21%, 250=1.43%, 500=0.26%, 1000=5.60% lat (msec) : 2=9.90%, 4=18.36%, 10=18.36%, 20=41.67%, 50=2.21% cpu : usr=0.16%, sys=21.20%, ctx=2626, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12192: Tue Apr 16 17:33:14 2024 read: IOPS=56, BW=908KiB/s (930kB/s)(5744KiB/6323msec) clat (usec): min=49, max=28650, avg=12453.30, stdev=4812.45 lat (usec): min=49, max=28651, avg=12454.09, stdev=4812.57 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 78], 10.00th=[ 7635], 20.00th=[10028], | 30.00th=[11338], 40.00th=[12387], 50.00th=[12911], 60.00th=[13829], | 70.00th=[14615], 80.00th=[15795], 90.00th=[17433], 95.00th=[19006], | 99.00th=[22676], 99.50th=[26084], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 608, max= 1088, per=22.81%, avg=856.00, stdev=138.56, samples=12 iops : min= 38, max= 68, avg=53.50, stdev= 8.66, samples=12 write: IOPS=64, BW=1035KiB/s (1060kB/s)(6544KiB/6323msec) clat (usec): min=667, max=27648, avg=4505.14, stdev=4395.94 lat (usec): min=669, max=27649, avg=4506.78, stdev=4395.75 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 824], 10.00th=[ 914], 20.00th=[ 1106], | 30.00th=[ 1467], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3818], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[10421], 95.00th=[15139], | 99.00th=[19792], 99.50th=[20579], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 608, max= 1696, per=24.52%, avg=968.00, stdev=274.30, samples=12 iops : min= 38, max= 106, avg=60.50, stdev=17.14, samples=12 lat (usec) : 50=0.13%, 100=2.86%, 250=0.78%, 750=0.91%, 1000=7.16% lat (msec) : 2=9.64%, 4=15.76%, 10=19.79%, 20=41.41%, 50=1.56% cpu : usr=0.06%, sys=21.43%, ctx=2640, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12193: Tue Apr 16 17:33:14 2024 read: IOPS=61, BW=980KiB/s (1004kB/s)(6256KiB/6382msec) clat (usec): min=53, max=26025, avg=12116.55, stdev=4832.05 lat (usec): min=53, max=26081, avg=12117.46, stdev=4832.50 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 83], 10.00th=[ 7570], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[11994], 50.00th=[12649], 60.00th=[13435], | 70.00th=[14484], 80.00th=[15270], 90.00th=[16581], 95.00th=[19268], | 99.00th=[22676], 99.50th=[23200], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 766, max= 1213, per=24.06%, avg=903.00, stdev=136.84, samples=12 iops : min= 47, max= 75, avg=56.08, stdev= 8.55, samples=12 write: IOPS=59, BW=945KiB/s (968kB/s)(6032KiB/6382msec) clat (usec): min=673, max=24498, avg=4335.17, stdev=4110.90 lat (usec): min=674, max=24500, avg=4336.49, stdev=4110.99 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 857], 10.00th=[ 930], 20.00th=[ 1123], | 30.00th=[ 1713], 40.00th=[ 2671], 50.00th=[ 3163], 60.00th=[ 3720], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 9896], 95.00th=[14615], | 99.00th=[19792], 99.50th=[21103], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 640, max= 1216, per=22.40%, avg=884.42, stdev=157.57, samples=12 iops : min= 40, max= 76, avg=54.92, stdev= 9.90, samples=12 lat (usec) : 100=3.26%, 250=1.04%, 750=0.65%, 1000=5.99% lat (msec) : 2=9.24%, 4=16.28%, 10=18.88%, 20=42.58%, 50=2.08% cpu : usr=0.13%, sys=21.60%, ctx=2758, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12194: Tue Apr 16 17:33:14 2024 read: IOPS=59, BW=952KiB/s (975kB/s)(6032KiB/6333msec) clat (usec): min=52, max=34745, avg=12492.26, stdev=4951.07 lat (usec): min=52, max=34746, avg=12493.21, stdev=4951.38 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 103], 10.00th=[ 7701], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[11994], 50.00th=[12911], 60.00th=[13698], | 70.00th=[14615], 80.00th=[15533], 90.00th=[17957], 95.00th=[19530], | 99.00th=[23462], 99.50th=[32375], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 768, max= 1184, per=24.08%, avg=903.67, stdev=123.16, samples=12 iops : min= 48, max= 74, avg=56.33, stdev= 7.82, samples=12 write: IOPS=61, BW=988KiB/s (1012kB/s)(6256KiB/6333msec) clat (usec): min=694, max=23433, avg=4119.69, stdev=3813.97 lat (usec): min=695, max=23434, avg=4121.98, stdev=3816.43 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 840], 10.00th=[ 914], 20.00th=[ 1139], | 30.00th=[ 1532], 40.00th=[ 2671], 50.00th=[ 3195], 60.00th=[ 3621], | 70.00th=[ 4359], 80.00th=[ 5866], 90.00th=[ 9241], 95.00th=[13829], | 99.00th=[18220], 99.50th=[20841], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 672, max= 1376, per=23.97%, avg=946.33, stdev=174.52, samples=12 iops : min= 42, max= 86, avg=59.00, stdev=10.95, samples=12 lat (usec) : 100=2.21%, 250=1.04%, 500=0.26%, 750=0.26%, 1000=7.55% lat (msec) : 2=9.51%, 4=15.89%, 10=21.61%, 20=39.45%, 50=2.21% cpu : usr=0.14%, sys=21.30%, ctx=2713, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3753KiB/s (3843kB/s), 908KiB/s-980KiB/s (930kB/s-1004kB/s), io=23.4MiB (24.5MB), run=6323-6382msec WRITE: bw=3949KiB/s (4043kB/s), 945KiB/s-1035KiB/s (968kB/s-1060kB/s), io=24.6MiB (25.8MB), run=6323-6382msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12206: Tue Apr 16 17:33:18 2024 read: IOPS=2, BW=2847KiB/s (2915kB/s)(7168KiB/2518msec) clat (usec): min=594, max=759808, avg=334565.66, stdev=295559.71 lat (usec): min=595, max=759809, avg=334566.88, stdev=295559.50 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 54264], | 30.00th=[ 55313], 40.00th=[ 55313], 50.00th=[455082], 60.00th=[509608], | 70.00th=[509608], 80.00th=[509608], 90.00th=[759170], 95.00th=[759170], | 99.00th=[759170], 99.50th=[759170], 99.90th=[759170], 99.95th=[759170], | 99.99th=[759170] bw ( KiB/s): min= 2043, max= 5647, per=36.21%, avg=2945.25, stdev=1801.17, samples=4 iops : min= 1, max= 5, avg= 2.25, stdev= 1.89, samples=4 write: IOPS=1, BW=2033KiB/s (2082kB/s)(5120KiB/2518msec) clat (usec): min=24631, max=43555, avg=32450.40, stdev=8396.29 lat (usec): min=24649, max=43636, avg=32515.48, stdev=8413.49 clat percentiles (usec): | 1.00th=[24511], 5.00th=[24511], 10.00th=[24511], 20.00th=[24511], | 30.00th=[27395], 40.00th=[27395], 50.00th=[27395], 60.00th=[27395], | 70.00th=[39060], 80.00th=[39060], 90.00th=[43779], 95.00th=[43779], | 99.00th=[43779], 99.50th=[43779], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 2043, max= 5647, per=33.77%, avg=3845.00, stdev=2548.41, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=41.67%, 100=16.67%, 500=8.33%, 750=16.67%, 1000=8.33% cpu : usr=0.04%, sys=23.68%, ctx=1180, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12208: Tue Apr 16 17:33:18 2024 read: IOPS=1, BW=1196KiB/s (1224kB/s)(2048KiB/1713msec) clat (usec): min=15148, max=83053, avg=49101.18, stdev=48015.78 lat (usec): min=15149, max=83054, avg=49102.15, stdev=48016.28 clat percentiles (usec): | 1.00th=[15139], 5.00th=[15139], 10.00th=[15139], 20.00th=[15139], | 30.00th=[15139], 40.00th=[15139], 50.00th=[15139], 60.00th=[83362], | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 2048, max= 2048, per=25.18%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5978KiB/s (6121kB/s)(10.0MiB/1713msec) clat (msec): min=28, max=743, avg=160.26, stdev=243.75 lat (msec): min=28, max=743, avg=160.31, stdev=243.76 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 36], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 51], 60.00th=[ 57], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 468], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 4096, max=12288, per=71.95%, avg=8192.00, stdev=5792.62, samples=2 iops : min= 4, max= 12, avg= 8.00, stdev= 5.66, samples=2 lat (msec) : 20=8.33%, 50=33.33%, 100=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=19.45%, ctx=160, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12210: Tue Apr 16 17:33:18 2024 read: IOPS=3, BW=3692KiB/s (3781kB/s)(6144KiB/1664msec) clat (usec): min=735, max=493345, avg=130110.60, stdev=182167.11 lat (usec): min=736, max=493346, avg=130112.02, stdev=182166.71 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 734], 10.00th=[ 734], 20.00th=[ 20317], | 30.00th=[ 20317], 40.00th=[ 83362], 50.00th=[ 83362], 60.00th=[ 90702], | 70.00th=[ 91751], 80.00th=[ 91751], 90.00th=[492831], 95.00th=[492831], | 99.00th=[492831], 99.50th=[492831], 99.90th=[492831], 99.95th=[492831], | 99.99th=[492831] bw ( KiB/s): min= 4087, max= 4096, per=50.31%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3692KiB/s (3781kB/s)(6144KiB/1664msec) clat (msec): min=26, max=632, avg=144.74, stdev=240.15 lat (msec): min=27, max=633, avg=144.83, stdev=240.14 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 56], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 2043, max= 4096, per=29.96%, avg=3411.67, stdev=1185.30, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.06%, sys=21.71%, ctx=418, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12212: Tue Apr 16 17:33:18 2024 read: IOPS=3, BW=3870KiB/s (3963kB/s)(5120KiB/1323msec) clat (usec): min=406, max=314175, avg=91135.80, stdev=132153.74 lat (usec): min=406, max=314177, avg=91136.78, stdev=132154.07 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 408], | 30.00th=[ 635], 40.00th=[ 635], 50.00th=[ 32637], 60.00th=[ 32637], | 70.00th=[107480], 80.00th=[107480], 90.00th=[312476], 95.00th=[312476], | 99.00th=[312476], 99.50th=[312476], 99.90th=[312476], 99.95th=[312476], | 99.99th=[312476] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5418KiB/s (5548kB/s)(7168KiB/1323msec) clat (msec): min=23, max=555, avg=123.61, stdev=191.44 lat (msec): min=23, max=555, avg=123.68, stdev=191.45 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 51], 60.00th=[ 62], | 70.00th=[ 62], 80.00th=[ 92], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 2048, max=10240, per=53.96%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=33.33%, 100=25.00%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.08%, sys=20.35%, ctx=376, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8133KiB/s (8329kB/s), 1196KiB/s-3870KiB/s (1224kB/s-3963kB/s), io=20.0MiB (20.0MB), run=1323-2518msec WRITE: bw=11.1MiB/s (11.7MB/s), 2033KiB/s-5978KiB/s (2082kB/s-6121kB/s), io=28.0MiB (29.4MB), run=1323-2518msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12205: Tue Apr 16 17:33:18 2024 read: IOPS=2, BW=2825KiB/s (2893kB/s)(7168KiB/2537msec) clat (msec): min=27, max=663, avg=328.52, stdev=274.51 lat (msec): min=27, max=663, avg=328.53, stdev=274.51 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 53], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 363], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 609], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 3771, per=30.71%, avg=2478.75, stdev=861.50, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 write: IOPS=1, BW=2018KiB/s (2067kB/s)(5120KiB/2537msec) clat (usec): min=20279, max=61477, avg=44678.00, stdev=15125.01 lat (usec): min=20316, max=61557, avg=44736.04, stdev=15140.75 clat percentiles (usec): | 1.00th=[20317], 5.00th=[20317], 10.00th=[20317], 20.00th=[20317], | 30.00th=[44827], 40.00th=[44827], 50.00th=[45876], 60.00th=[45876], | 70.00th=[50594], 80.00th=[50594], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 1885, max= 2048, per=17.40%, avg=1966.50, stdev=115.26, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 500=8.33%, 750=25.00% cpu : usr=0.00%, sys=6.19%, ctx=58, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12207: Tue Apr 16 17:33:18 2024 read: IOPS=1, BW=1160KiB/s (1188kB/s)(2048KiB/1766msec) clat (msec): min=46, max=127, avg=86.79, stdev=57.63 lat (msec): min=46, max=127, avg=86.79, stdev=57.63 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 2031, max= 2048, per=25.27%, avg=2039.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5798KiB/s (5938kB/s)(10.0MiB/1766msec) clat (msec): min=27, max=681, avg=158.41, stdev=223.42 lat (msec): min=27, max=681, avg=158.49, stdev=223.43 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 39], | 30.00th=[ 49], 40.00th=[ 50], 50.00th=[ 61], 60.00th=[ 70], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 456], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2043, max=10158, per=48.07%, avg=5432.33, stdev=4219.32, samples=3 iops : min= 1, max= 9, avg= 4.67, stdev= 4.04, samples=3 lat (msec) : 50=41.67%, 100=33.33%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=11.22%, ctx=92, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12209: Tue Apr 16 17:33:18 2024 read: IOPS=3, BW=3471KiB/s (3554kB/s)(6144KiB/1770msec) clat (msec): min=23, max=645, avg=149.38, stdev=243.84 lat (msec): min=23, max=645, avg=149.38, stdev=243.83 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 56], | 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 8192, per=63.43%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=3, BW=3471KiB/s (3554kB/s)(6144KiB/1770msec) clat (msec): min=34, max=608, avg=141.60, stdev=229.30 lat (msec): min=35, max=608, avg=141.68, stdev=229.29 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 38], | 30.00th=[ 38], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 55], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2048, max= 6144, per=36.24%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=50.00%, 100=33.33%, 750=16.67% cpu : usr=0.06%, sys=8.48%, ctx=199, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12211: Tue Apr 16 17:33:18 2024 read: IOPS=2, BW=2977KiB/s (3048kB/s)(5120KiB/1720msec) clat (msec): min=23, max=630, avg=158.74, stdev=264.14 lat (msec): min=23, max=630, avg=158.74, stdev=264.14 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4167KiB/s (4267kB/s)(7168KiB/1720msec) clat (msec): min=28, max=614, avg=132.09, stdev=214.29 lat (msec): min=28, max=614, avg=132.17, stdev=214.30 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 34], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 55], 80.00th=[ 105], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 2048, max= 6144, per=42.21%, avg=4770.33, stdev=2357.64, samples=3 iops : min= 2, max= 6, avg= 4.33, stdev= 2.08, samples=3 lat (msec) : 50=50.00%, 100=25.00%, 250=8.33%, 750=16.67% cpu : usr=0.12%, sys=7.21%, ctx=84, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8073KiB/s (8266kB/s), 1160KiB/s-3471KiB/s (1188kB/s-3554kB/s), io=20.0MiB (20.0MB), run=1720-2537msec WRITE: bw=11.0MiB/s (11.6MB/s), 2018KiB/s-5798KiB/s (2067kB/s-5938kB/s), io=28.0MiB (29.4MB), run=1720-2537msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12219: Tue Apr 16 17:33:21 2024 read: IOPS=1, BW=5968KiB/s (6111kB/s)(12.0MiB/2059msec) clat (msec): min=239, max=1305, avg=683.42, stdev=555.05 lat (msec): min=239, max=1305, avg=683.43, stdev=555.05 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=48.40%, ctx=748, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12220: Tue Apr 16 17:33:21 2024 read: IOPS=0, BW=1853KiB/s (1897kB/s)(4096KiB/2211msec) clat (nsec): min=234888k, max=234888k, avg=234887813.00, stdev= 0.00 lat (nsec): min=234889k, max=234889k, avg=234888888.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3705KiB/s (3794kB/s)(8192KiB/2211msec) clat (msec): min=121, max=1849, avg=985.48, stdev=1221.88 lat (msec): min=121, max=1849, avg=985.74, stdev=1222.08 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] lat (msec) : 250=66.67% cpu : usr=0.00%, sys=15.34%, ctx=167, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12221: Tue Apr 16 17:33:21 2024 write: IOPS=1, BW=5170KiB/s (5294kB/s)(12.0MiB/2377msec) clat (msec): min=768, max=836, avg=791.64, stdev=39.08 lat (msec): min=768, max=837, avg=791.99, stdev=39.05 clat percentiles (msec): | 1.00th=[ 768], 5.00th=[ 768], 10.00th=[ 768], 20.00th=[ 768], | 30.00th=[ 768], 40.00th=[ 768], 50.00th=[ 768], 60.00th=[ 768], | 70.00th=[ 835], 80.00th=[ 835], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7543, max= 8192, per=57.07%, avg=7867.50, stdev=458.91, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=13.05%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12222: Tue Apr 16 17:33:21 2024 write: IOPS=2, BW=11.0MiB/s (11.6MB/s)(12.0MiB/1088msec) clat (msec): min=148, max=631, avg=359.14, stdev=247.16 lat (msec): min=148, max=631, avg=359.44, stdev=247.11 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8192, max= 8192, per=59.43%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=28.70%, ctx=24, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7410KiB/s (7588kB/s), 1853KiB/s-5968KiB/s (1897kB/s-6111kB/s), io=16.0MiB (16.8MB), run=2059-2211msec WRITE: bw=13.5MiB/s (14.1MB/s), 3705KiB/s-11.0MiB/s (3794kB/s-11.6MB/s), io=32.0MiB (33.6MB), run=1088-2377msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12224: Tue Apr 16 17:33:21 2024 read: IOPS=1, BW=4981KiB/s (5100kB/s)(12.0MiB/2467msec) clat (msec): min=490, max=1087, avg=821.74, stdev=303.84 lat (msec): min=490, max=1087, avg=821.74, stdev=303.84 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 885], 50.00th=[ 885], 60.00th=[ 885], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=3.97%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12225: Tue Apr 16 17:33:21 2024 read: IOPS=0, BW=1699KiB/s (1740kB/s)(4096KiB/2411msec) clat (nsec): min=915530k, max=915530k, avg=915529907.00, stdev= 0.00 lat (nsec): min=915533k, max=915533k, avg=915532637.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 919], 5.00th=[ 919], 10.00th=[ 919], 20.00th=[ 919], | 30.00th=[ 919], 40.00th=[ 919], 50.00th=[ 919], 60.00th=[ 919], | 70.00th=[ 919], 80.00th=[ 919], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3398KiB/s (3479kB/s)(8192KiB/2411msec) clat (msec): min=187, max=1298, avg=742.90, stdev=786.00 lat (msec): min=187, max=1298, avg=743.16, stdev=785.97 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 1301], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=4.69%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12226: Tue Apr 16 17:33:21 2024 write: IOPS=1, BW=4796KiB/s (4911kB/s)(12.0MiB/2562msec) clat (msec): min=621, max=1079, avg=848.72, stdev=228.69 lat (msec): min=622, max=1079, avg=849.06, stdev=228.67 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 844], 50.00th=[ 844], 60.00th=[ 844], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 8192, max= 8192, per=64.05%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.12%, sys=6.72%, ctx=57, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12227: Tue Apr 16 17:33:21 2024 write: IOPS=1, BW=5059KiB/s (5180kB/s)(12.0MiB/2429msec) clat (msec): min=158, max=2058, avg=808.78, stdev=1082.34 lat (msec): min=159, max=2058, avg=808.98, stdev=1082.32 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] lat (msec) : 250=66.67% cpu : usr=0.04%, sys=7.66%, ctx=90, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6641KiB/s (6801kB/s), 1699KiB/s-4981KiB/s (1740kB/s-5100kB/s), io=16.0MiB (16.8MB), run=2411-2467msec WRITE: bw=12.5MiB/s (13.1MB/s), 3398KiB/s-5059KiB/s (3479kB/s-5180kB/s), io=32.0MiB (33.6MB), run=2411-2562msec