== sanity test 398b: DIO and buffer IO race ============== 13:14:11 (1713287651) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.03124 s, 48.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30380: Tue Apr 16 13:14:26 2024 read: IOPS=123, BW=495KiB/s (506kB/s)(5992KiB/12115msec) clat (usec): min=46, max=163379, avg=3713.17, stdev=5211.48 lat (usec): min=46, max=163380, avg=3713.86, stdev=5211.47 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 70], 10.00th=[ 75], 20.00th=[ 285], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 2966], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5669], 90.00th=[ 7242], 95.00th=[ 8848], | 99.00th=[ 14353], 99.50th=[ 19268], 99.90th=[ 36963], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 232, max= 816, per=25.16%, avg=492.67, stdev=106.46, samples=24 iops : min= 58, max= 204, avg=123.08, stdev=26.64, samples=24 write: IOPS=129, BW=520KiB/s (532kB/s)(6296KiB/12115msec) clat (usec): min=458, max=33122, avg=4092.08, stdev=3808.15 lat (usec): min=458, max=33123, avg=4092.93, stdev=3808.33 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 676], | 30.00th=[ 906], 40.00th=[ 1942], 50.00th=[ 3294], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 6849], 90.00th=[ 9241], 95.00th=[11600], | 99.00th=[15270], 99.50th=[18482], 99.90th=[30278], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 296, max= 760, per=25.23%, avg=517.21, stdev=102.73, samples=24 iops : min= 74, max= 190, avg=129.21, stdev=25.70, samples=24 lat (usec) : 50=0.23%, 100=7.55%, 250=1.86%, 500=0.72%, 750=12.40% lat (usec) : 1000=3.45% lat (msec) : 2=5.40%, 4=29.85%, 10=32.81%, 20=5.44%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=24.83%, ctx=6201, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30381: Tue Apr 16 13:14:26 2024 read: IOPS=122, BW=491KiB/s (503kB/s)(5940KiB/12091msec) clat (usec): min=46, max=23511, avg=3469.24, stdev=2795.81 lat (usec): min=46, max=23512, avg=3469.85, stdev=2795.85 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 71], 10.00th=[ 75], 20.00th=[ 128], | 30.00th=[ 2245], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4293], 80.00th=[ 5473], 90.00th=[ 7242], 95.00th=[ 8356], | 99.00th=[11731], 99.50th=[12911], 99.90th=[22152], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 296, max= 632, per=24.65%, avg=482.74, stdev=81.03, samples=23 iops : min= 74, max= 158, avg=120.48, stdev=20.25, samples=23 write: IOPS=131, BW=525KiB/s (538kB/s)(6348KiB/12091msec) clat (usec): min=476, max=191908, avg=4302.11, stdev=6179.51 lat (usec): min=477, max=191909, avg=4302.87, stdev=6179.54 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 693], | 30.00th=[ 996], 40.00th=[ 2343], 50.00th=[ 3458], 60.00th=[ 4228], | 70.00th=[ 5276], 80.00th=[ 6915], 90.00th=[ 9503], 95.00th=[ 11731], | 99.00th=[ 16319], 99.50th=[ 19530], 99.90th=[ 44827], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 352, max= 776, per=24.99%, avg=512.26, stdev=90.92, samples=23 iops : min= 88, max= 194, avg=127.87, stdev=22.78, samples=23 lat (usec) : 50=0.03%, 100=8.33%, 250=1.86%, 500=0.78%, 750=11.39% lat (usec) : 1000=3.71% lat (msec) : 2=5.47%, 4=30.05%, 10=33.04%, 20=5.05%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=24.45%, ctx=6194, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30382: Tue Apr 16 13:14:26 2024 read: IOPS=122, BW=492KiB/s (504kB/s)(6028KiB/12258msec) clat (usec): min=45, max=191809, avg=3665.43, stdev=5573.46 lat (usec): min=45, max=191810, avg=3666.22, stdev=5573.41 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 71], 10.00th=[ 75], 20.00th=[ 172], | 30.00th=[ 2311], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 7242], 95.00th=[ 8455], | 99.00th=[ 11207], 99.50th=[ 15270], 99.90th=[ 21365], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 232, max= 728, per=24.69%, avg=483.33, stdev=104.85, samples=24 iops : min= 58, max= 182, avg=120.79, stdev=26.16, samples=24 write: IOPS=127, BW=511KiB/s (523kB/s)(6260KiB/12258msec) clat (usec): min=465, max=40147, avg=4231.35, stdev=4138.75 lat (usec): min=465, max=40148, avg=4232.17, stdev=4138.72 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 685], | 30.00th=[ 922], 40.00th=[ 2180], 50.00th=[ 3458], 60.00th=[ 4359], | 70.00th=[ 5342], 80.00th=[ 7046], 90.00th=[ 9241], 95.00th=[11731], | 99.00th=[18220], 99.50th=[22676], 99.90th=[32900], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 312, max= 680, per=24.40%, avg=500.25, stdev=100.62, samples=24 iops : min= 78, max= 170, avg=125.04, stdev=25.14, samples=24 lat (usec) : 50=0.39%, 100=7.71%, 250=1.99%, 500=0.46%, 750=11.78% lat (usec) : 1000=4.13% lat (msec) : 2=5.01%, 4=28.74%, 10=35.22%, 20=4.07%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=24.73%, ctx=6241, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30383: Tue Apr 16 13:14:26 2024 read: IOPS=128, BW=515KiB/s (527kB/s)(6052KiB/11750msec) clat (usec): min=47, max=165247, avg=3707.22, stdev=5252.17 lat (usec): min=47, max=165248, avg=3708.46, stdev=5252.19 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 71], 10.00th=[ 76], 20.00th=[ 1745], | 30.00th=[ 2278], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 5669], 90.00th=[ 7242], 95.00th=[ 8848], | 99.00th=[ 13042], 99.50th=[ 16057], 99.90th=[ 58459], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 280, max= 776, per=26.30%, avg=515.04, stdev=102.90, samples=23 iops : min= 70, max= 194, avg=128.74, stdev=25.70, samples=23 write: IOPS=132, BW=531KiB/s (543kB/s)(6236KiB/11750msec) clat (usec): min=478, max=37994, avg=3870.83, stdev=3807.37 lat (usec): min=479, max=37995, avg=3871.52, stdev=3807.37 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 660], | 30.00th=[ 832], 40.00th=[ 1352], 50.00th=[ 2900], 60.00th=[ 3916], | 70.00th=[ 5080], 80.00th=[ 6521], 90.00th=[ 9110], 95.00th=[11338], | 99.00th=[15926], 99.50th=[17171], 99.90th=[28181], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 328, max= 704, per=25.92%, avg=531.39, stdev=97.62, samples=23 iops : min= 82, max= 176, avg=132.83, stdev=24.37, samples=23 lat (usec) : 50=0.20%, 100=7.49%, 250=2.02%, 500=0.46%, 750=12.99% lat (usec) : 1000=4.82% lat (msec) : 2=5.66%, 4=29.46%, 10=31.54%, 20=5.14%, 50=0.16% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=24.97%, ctx=6161, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1959KiB/s (2006kB/s), 491KiB/s-515KiB/s (503kB/s-527kB/s), io=23.4MiB (24.6MB), run=11750-12258msec WRITE: bw=2051KiB/s (2100kB/s), 511KiB/s-531KiB/s (523kB/s-543kB/s), io=24.6MiB (25.7MB), run=11750-12258msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30376: Tue Apr 16 13:14:55 2024 read: IOPS=35, BW=144KiB/s (147kB/s)(5992KiB/41702msec) clat (usec): min=1552, max=198213, avg=4512.90, stdev=5847.98 lat (usec): min=1553, max=198214, avg=4513.64, stdev=5848.03 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1893], 10.00th=[ 2073], 20.00th=[ 2409], | 30.00th=[ 2802], 40.00th=[ 3425], 50.00th=[ 3752], 60.00th=[ 4047], | 70.00th=[ 4490], 80.00th=[ 5800], 90.00th=[ 7308], 95.00th=[ 9372], | 99.00th=[ 14746], 99.50th=[ 16712], 99.90th=[ 49021], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 48, max= 247, per=24.95%, avg=143.45, stdev=48.66, samples=83 iops : min= 12, max= 61, avg=35.72, stdev=12.16, samples=83 write: IOPS=37, BW=151KiB/s (155kB/s)(6296KiB/41702msec) clat (usec): min=12169, max=65123, avg=22125.29, stdev=6824.59 lat (usec): min=12170, max=65123, avg=22126.25, stdev=6824.51 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13960], 10.00th=[14746], 20.00th=[16057], | 30.00th=[17695], 40.00th=[19530], 50.00th=[21365], 60.00th=[22938], | 70.00th=[24249], 80.00th=[26608], 90.00th=[31065], 95.00th=[35390], | 99.00th=[44303], 99.50th=[46924], 99.90th=[55313], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 40, max= 192, per=24.98%, avg=150.40, stdev=33.83, samples=83 iops : min= 10, max= 48, avg=37.46, stdev= 8.44, samples=83 lat (msec) : 2=3.84%, 4=24.51%, 10=18.46%, 20=23.34%, 50=29.69% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.08%, sys=5.02%, ctx=4041, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30377: Tue Apr 16 13:14:55 2024 read: IOPS=35, BW=143KiB/s (146kB/s)(5940KiB/41593msec) clat (usec): min=1573, max=170750, avg=4569.06, stdev=5243.30 lat (usec): min=1574, max=170752, avg=4569.92, stdev=5243.29 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2114], 20.00th=[ 2442], | 30.00th=[ 2835], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4080], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 7504], 95.00th=[ 9503], | 99.00th=[ 16057], 99.50th=[ 18482], 99.90th=[ 39060], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 15, max= 287, per=24.83%, avg=142.76, stdev=57.36, samples=82 iops : min= 3, max= 71, avg=35.63, stdev=14.33, samples=82 write: IOPS=38, BW=153KiB/s (156kB/s)(6348KiB/41593msec) clat (usec): min=12003, max=66733, avg=21860.70, stdev=6982.68 lat (usec): min=12003, max=66734, avg=21861.59, stdev=6982.65 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13829], 10.00th=[14615], 20.00th=[15926], | 30.00th=[17171], 40.00th=[18744], 50.00th=[20579], 60.00th=[22414], | 70.00th=[23987], 80.00th=[26346], 90.00th=[30802], 95.00th=[35914], | 99.00th=[43779], 99.50th=[49546], 99.90th=[58459], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 55, max= 208, per=25.20%, avg=151.73, stdev=34.19, samples=82 iops : min= 13, max= 52, avg=37.88, stdev= 8.56, samples=82 lat (msec) : 2=3.65%, 4=23.76%, 10=18.75%, 20=26.27%, 50=27.34% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.07%, sys=5.00%, ctx=4037, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30378: Tue Apr 16 13:14:55 2024 read: IOPS=36, BW=145KiB/s (149kB/s)(6028KiB/41441msec) clat (usec): min=1578, max=30930, avg=4492.18, stdev=2818.81 lat (usec): min=1579, max=30931, avg=4492.89, stdev=2818.83 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2114], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3490], 50.00th=[ 3851], 60.00th=[ 4146], | 70.00th=[ 4621], 80.00th=[ 5997], 90.00th=[ 7832], 95.00th=[ 9765], | 99.00th=[14353], 99.50th=[18482], 99.90th=[28967], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 48, max= 272, per=25.32%, avg=145.61, stdev=46.92, samples=82 iops : min= 12, max= 68, avg=36.29, stdev=11.71, samples=82 write: IOPS=37, BW=151KiB/s (155kB/s)(6260KiB/41441msec) clat (msec): min=11, max=186, avg=22.08, stdev= 7.85 lat (msec): min=11, max=186, avg=22.08, stdev= 7.85 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 25], 80.00th=[ 27], 90.00th=[ 31], 95.00th=[ 36], | 99.00th=[ 44], 99.50th=[ 48], 99.90th=[ 58], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 48, max= 200, per=25.03%, avg=150.68, stdev=33.23, samples=82 iops : min= 12, max= 50, avg=37.56, stdev= 8.33, samples=82 lat (msec) : 2=3.45%, 4=24.02%, 10=19.27%, 20=23.80%, 50=29.23% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.10%, sys=5.00%, ctx=4000, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30379: Tue Apr 16 13:14:55 2024 read: IOPS=36, BW=147KiB/s (151kB/s)(6052KiB/41156msec) clat (usec): min=1556, max=49218, avg=4389.34, stdev=3199.00 lat (usec): min=1557, max=49219, avg=4390.06, stdev=3199.03 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1860], 10.00th=[ 2073], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 3392], 50.00th=[ 3720], 60.00th=[ 4047], | 70.00th=[ 4490], 80.00th=[ 5538], 90.00th=[ 7504], 95.00th=[ 9110], | 99.00th=[14222], 99.50th=[16188], 99.90th=[44827], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 39, max= 280, per=25.57%, avg=147.04, stdev=53.51, samples=82 iops : min= 9, max= 70, avg=36.70, stdev=13.41, samples=82 write: IOPS=37, BW=152KiB/s (155kB/s)(6236KiB/41156msec) clat (msec): min=12, max=185, avg=22.07, stdev= 7.87 lat (msec): min=12, max=185, avg=22.07, stdev= 7.87 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 25], 80.00th=[ 27], 90.00th=[ 32], 95.00th=[ 36], | 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 66], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 55, max= 200, per=25.12%, avg=151.23, stdev=33.81, samples=82 iops : min= 13, max= 50, avg=37.74, stdev= 8.49, samples=82 lat (msec) : 2=4.20%, 4=24.54%, 10=18.52%, 20=24.06%, 50=28.58% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.09%, sys=5.01%, ctx=3992, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=576KiB/s (590kB/s), 143KiB/s-147KiB/s (146kB/s-151kB/s), io=23.4MiB (24.6MB), run=41156-41702msec WRITE: bw=603KiB/s (617kB/s), 151KiB/s-153KiB/s (155kB/s-156kB/s), io=24.6MiB (25.7MB), run=41156-41702msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30427: Tue Apr 16 13:15:01 2024 read: IOPS=85, BW=1362KiB/s (1394kB/s)(5920KiB/4348msec) clat (usec): min=53, max=53147, avg=8591.03, stdev=5836.64 lat (usec): min=54, max=53148, avg=8591.61, stdev=5836.66 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 73], 10.00th=[ 75], 20.00th=[ 97], | 30.00th=[ 7767], 40.00th=[ 8848], 50.00th=[ 9896], 60.00th=[10552], | 70.00th=[11600], 80.00th=[12780], 90.00th=[14091], 95.00th=[15533], | 99.00th=[20841], 99.50th=[23725], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 1056, max= 1536, per=23.94%, avg=1308.00, stdev=171.42, samples=8 iops : min= 66, max= 96, avg=81.75, stdev=10.71, samples=8 write: IOPS=91, BW=1465KiB/s (1500kB/s)(6368KiB/4348msec) clat (usec): min=673, max=23276, avg=2920.77, stdev=3219.56 lat (usec): min=674, max=23277, avg=2921.83, stdev=3219.54 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 816], | 30.00th=[ 865], 40.00th=[ 938], 50.00th=[ 1434], 60.00th=[ 2638], | 70.00th=[ 3392], 80.00th=[ 4686], 90.00th=[ 6587], 95.00th=[ 8455], | 99.00th=[16450], 99.50th=[21365], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 1024, max= 1824, per=25.12%, avg=1444.00, stdev=253.38, samples=8 iops : min= 64, max= 114, avg=90.25, stdev=15.84, samples=8 lat (usec) : 100=10.03%, 250=1.56%, 750=3.52%, 1000=19.79% lat (msec) : 2=4.69%, 4=11.85%, 10=24.35%, 20=23.31%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.21%, sys=21.97%, ctx=1916, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30428: Tue Apr 16 13:15:01 2024 read: IOPS=83, BW=1332KiB/s (1364kB/s)(5744KiB/4311msec) clat (usec): min=51, max=20236, avg=8610.75, stdev=5064.39 lat (usec): min=51, max=20237, avg=8611.72, stdev=5064.49 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 75], 20.00th=[ 93], | 30.00th=[ 8160], 40.00th=[ 9241], 50.00th=[10159], 60.00th=[10814], | 70.00th=[11469], 80.00th=[12518], 90.00th=[13566], 95.00th=[14746], | 99.00th=[19268], 99.50th=[19792], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 960, max= 1600, per=24.24%, avg=1324.00, stdev=220.34, samples=8 iops : min= 60, max= 100, avg=82.75, stdev=13.77, samples=8 write: IOPS=94, BW=1518KiB/s (1554kB/s)(6544KiB/4311msec) clat (usec): min=680, max=79288, avg=2964.59, stdev=5527.61 lat (usec): min=681, max=79289, avg=2965.57, stdev=5527.59 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 807], | 30.00th=[ 857], 40.00th=[ 930], 50.00th=[ 1156], 60.00th=[ 2409], | 70.00th=[ 3163], 80.00th=[ 4080], 90.00th=[ 6063], 95.00th=[ 7832], | 99.00th=[15926], 99.50th=[33817], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 1120, max= 1888, per=25.68%, avg=1476.00, stdev=256.25, samples=8 iops : min= 70, max= 118, avg=92.25, stdev=16.02, samples=8 lat (usec) : 100=9.51%, 250=0.65%, 750=4.30%, 1000=20.44% lat (msec) : 2=6.25%, 4=11.33%, 10=22.27%, 20=24.74%, 50=0.26% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=21.62%, ctx=1933, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30430: Tue Apr 16 13:15:01 2024 read: IOPS=90, BW=1441KiB/s (1476kB/s)(6256KiB/4341msec) clat (usec): min=49, max=44141, avg=8249.37, stdev=5810.71 lat (usec): min=50, max=44142, avg=8249.93, stdev=5810.72 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 67], 10.00th=[ 74], 20.00th=[ 88], | 30.00th=[ 7767], 40.00th=[ 8979], 50.00th=[ 9503], 60.00th=[10290], | 70.00th=[11207], 80.00th=[11994], 90.00th=[13829], 95.00th=[15401], | 99.00th=[22676], 99.50th=[30278], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 896, max= 1984, per=25.84%, avg=1411.62, stdev=318.92, samples=8 iops : min= 56, max= 124, avg=88.12, stdev=19.96, samples=8 write: IOPS=86, BW=1390KiB/s (1423kB/s)(6032KiB/4341msec) clat (usec): min=674, max=67095, avg=2938.51, stdev=4448.48 lat (usec): min=675, max=67097, avg=2939.40, stdev=4448.51 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 725], 10.00th=[ 750], 20.00th=[ 807], | 30.00th=[ 848], 40.00th=[ 922], 50.00th=[ 1254], 60.00th=[ 2540], | 70.00th=[ 3130], 80.00th=[ 4424], 90.00th=[ 6587], 95.00th=[ 9110], | 99.00th=[17433], 99.50th=[21890], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 862, max= 2208, per=23.45%, avg=1347.75, stdev=403.10, samples=8 iops : min= 53, max= 138, avg=84.12, stdev=25.35, samples=8 lat (usec) : 50=0.13%, 100=11.72%, 250=1.82%, 750=4.30%, 1000=18.23% lat (msec) : 2=4.30%, 4=11.46%, 10=24.48%, 20=22.66%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=21.45%, ctx=1938, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30432: Tue Apr 16 13:15:01 2024 read: IOPS=85, BW=1376KiB/s (1409kB/s)(6032KiB/4384msec) clat (usec): min=48, max=30525, avg=8535.40, stdev=5795.75 lat (usec): min=48, max=30525, avg=8535.95, stdev=5795.76 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 83], | 30.00th=[ 7832], 40.00th=[ 8979], 50.00th=[ 9765], 60.00th=[10683], | 70.00th=[11600], 80.00th=[12649], 90.00th=[14091], 95.00th=[16450], | 99.00th=[24249], 99.50th=[27919], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 928, max= 1696, per=24.53%, avg=1340.00, stdev=224.61, samples=8 iops : min= 58, max= 106, avg=83.75, stdev=14.04, samples=8 write: IOPS=89, BW=1427KiB/s (1461kB/s)(6256KiB/4384msec) clat (usec): min=668, max=70497, avg=2960.96, stdev=4752.02 lat (usec): min=668, max=70499, avg=2961.84, stdev=4752.04 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 775], 20.00th=[ 807], | 30.00th=[ 873], 40.00th=[ 947], 50.00th=[ 1369], 60.00th=[ 2573], | 70.00th=[ 3163], 80.00th=[ 4359], 90.00th=[ 6325], 95.00th=[ 7832], | 99.00th=[23725], 99.50th=[31327], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 1152, max= 2080, per=24.70%, avg=1420.00, stdev=336.46, samples=8 iops : min= 72, max= 130, avg=88.75, stdev=21.03, samples=8 lat (usec) : 50=0.26%, 100=11.33%, 250=1.17%, 750=2.34%, 1000=20.70% lat (msec) : 2=4.56%, 4=12.37%, 10=22.27%, 20=23.31%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=21.06%, ctx=1961, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5464KiB/s (5595kB/s), 1332KiB/s-1441KiB/s (1364kB/s-1476kB/s), io=23.4MiB (24.5MB), run=4311-4384msec WRITE: bw=5748KiB/s (5886kB/s), 1390KiB/s-1518KiB/s (1423kB/s-1554kB/s), io=24.6MiB (25.8MB), run=4311-4384msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30426: Tue Apr 16 13:15:07 2024 read: IOPS=35, BW=567KiB/s (581kB/s)(5920KiB/10439msec) clat (usec): min=2017, max=51839, avg=4273.56, stdev=4461.42 lat (usec): min=2017, max=51840, avg=4274.21, stdev=4461.43 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2278], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3261], 80.00th=[ 5211], 90.00th=[ 8455], 95.00th=[11994], | 99.00th=[21627], 99.50th=[23725], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 320, max= 1024, per=24.99%, avg=558.30, stdev=179.06, samples=20 iops : min= 20, max= 64, avg=34.85, stdev=11.19, samples=20 write: IOPS=38, BW=610KiB/s (625kB/s)(6368KiB/10439msec) clat (usec): min=12596, max=65191, avg=22233.86, stdev=6932.40 lat (usec): min=12598, max=65192, avg=22235.15, stdev=6932.44 clat percentiles (usec): | 1.00th=[13173], 5.00th=[13698], 10.00th=[14484], 20.00th=[16057], | 30.00th=[18220], 40.00th=[20055], 50.00th=[21627], 60.00th=[22938], | 70.00th=[24249], 80.00th=[26084], 90.00th=[30802], 95.00th=[34866], | 99.00th=[46400], 99.50th=[60556], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 384, max= 768, per=25.66%, avg=603.10, stdev=142.00, samples=20 iops : min= 24, max= 48, avg=37.65, stdev= 8.85, samples=20 lat (msec) : 4=36.07%, 10=7.94%, 20=24.22%, 50=31.25%, 100=0.52% cpu : usr=0.09%, sys=5.09%, ctx=884, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30429: Tue Apr 16 13:15:07 2024 read: IOPS=33, BW=536KiB/s (549kB/s)(5744KiB/10720msec) clat (usec): min=1919, max=24190, avg=3934.06, stdev=3378.48 lat (usec): min=1920, max=24191, avg=3934.70, stdev=3378.49 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2114], 10.00th=[ 2147], 20.00th=[ 2245], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2540], 60.00th=[ 2802], | 70.00th=[ 3163], 80.00th=[ 4752], 90.00th=[ 7308], 95.00th=[11731], | 99.00th=[19006], 99.50th=[23725], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 160, max= 928, per=24.00%, avg=536.14, stdev=206.78, samples=21 iops : min= 10, max= 58, avg=33.43, stdev=12.90, samples=21 write: IOPS=38, BW=610KiB/s (625kB/s)(6544KiB/10720msec) clat (msec): min=12, max=139, avg=22.74, stdev= 9.05 lat (msec): min=12, max=139, avg=22.74, stdev= 9.05 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 25], 80.00th=[ 28], 90.00th=[ 33], 95.00th=[ 37], | 99.00th=[ 43], 99.50th=[ 48], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 352, max= 828, per=25.86%, avg=607.71, stdev=147.34, samples=21 iops : min= 22, max= 51, avg=37.90, stdev= 9.14, samples=21 lat (msec) : 2=0.26%, 4=35.29%, 10=8.20%, 20=24.74%, 50=31.25% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.09%, sys=4.88%, ctx=864, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30431: Tue Apr 16 13:15:07 2024 read: IOPS=39, BW=625KiB/s (640kB/s)(6256KiB/10016msec) clat (usec): min=1898, max=23346, avg=3688.37, stdev=3043.06 lat (usec): min=1899, max=23346, avg=3688.99, stdev=3043.07 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2278], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2704], | 70.00th=[ 3032], 80.00th=[ 3884], 90.00th=[ 6194], 95.00th=[11338], | 99.00th=[18744], 99.50th=[19792], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 288, max= 1152, per=28.00%, avg=625.60, stdev=222.55, samples=20 iops : min= 18, max= 72, avg=39.10, stdev=13.91, samples=20 write: IOPS=37, BW=602KiB/s (617kB/s)(6032KiB/10016msec) clat (usec): min=12753, max=54349, avg=22720.22, stdev=6984.02 lat (usec): min=12754, max=54352, avg=22721.49, stdev=6984.09 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13960], 10.00th=[14746], 20.00th=[16450], | 30.00th=[18482], 40.00th=[20317], 50.00th=[21627], 60.00th=[23462], | 70.00th=[25035], 80.00th=[27132], 90.00th=[32637], 95.00th=[35914], | 99.00th=[44827], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 384, max= 800, per=25.60%, avg=601.60, stdev=139.14, samples=20 iops : min= 24, max= 50, avg=37.60, stdev= 8.70, samples=20 lat (msec) : 2=0.26%, 4=40.49%, 10=7.16%, 20=22.01%, 50=29.82% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=5.21%, ctx=854, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30433: Tue Apr 16 13:15:07 2024 read: IOPS=36, BW=577KiB/s (590kB/s)(6032KiB/10462msec) clat (usec): min=1915, max=28307, avg=3710.03, stdev=3032.26 lat (usec): min=1915, max=28308, avg=3710.67, stdev=3032.24 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2278], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2704], | 70.00th=[ 3064], 80.00th=[ 4015], 90.00th=[ 7046], 95.00th=[10552], | 99.00th=[13960], 99.50th=[23462], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 288, max= 864, per=25.27%, avg=564.60, stdev=173.95, samples=20 iops : min= 18, max= 54, avg=35.20, stdev=10.80, samples=20 write: IOPS=37, BW=598KiB/s (612kB/s)(6256KiB/10462msec) clat (usec): min=12897, max=76363, avg=23161.89, stdev=7049.79 lat (usec): min=12898, max=76365, avg=23163.10, stdev=7049.88 clat percentiles (usec): | 1.00th=[13173], 5.00th=[14222], 10.00th=[15664], 20.00th=[17695], | 30.00th=[19006], 40.00th=[21103], 50.00th=[22152], 60.00th=[23200], | 70.00th=[25297], 80.00th=[27132], 90.00th=[31851], 95.00th=[35390], | 99.00th=[47973], 99.50th=[51643], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 352, max= 768, per=25.11%, avg=590.20, stdev=126.77, samples=20 iops : min= 22, max= 48, avg=36.80, stdev= 7.86, samples=20 lat (msec) : 2=0.26%, 4=38.93%, 10=7.29%, 20=19.01%, 50=34.11% lat (msec) : 100=0.39% cpu : usr=0.07%, sys=4.97%, ctx=851, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2234KiB/s (2288kB/s), 536KiB/s-625KiB/s (549kB/s-640kB/s), io=23.4MiB (24.5MB), run=10016-10720msec WRITE: bw=2351KiB/s (2407kB/s), 598KiB/s-610KiB/s (612kB/s-625kB/s), io=24.6MiB (25.8MB), run=10016-10720msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30452: Tue Apr 16 13:15:11 2024 read: IOPS=2, BW=2169KiB/s (2222kB/s)(7168KiB/3304msec) clat (usec): min=460, max=1047.1k, avg=428800.03, stdev=492149.70 lat (usec): min=462, max=1047.1k, avg=428800.99, stdev=492149.84 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], | 20.00th=[ 38536], 30.00th=[ 48497], 40.00th=[ 48497], | 50.00th=[ 58983], 60.00th=[ 876610], 70.00th=[ 876610], | 80.00th=[ 935330], 90.00th=[1044382], 95.00th=[1044382], | 99.00th=[1044382], 99.50th=[1044382], 99.90th=[1044382], | 99.95th=[1044382], 99.99th=[1044382] bw ( KiB/s): min= 2048, max= 7062, per=61.39%, avg=3719.33, stdev=2894.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1550KiB/s (1587kB/s)(5120KiB/3304msec) clat (msec): min=16, max=184, avg=59.31, stdev=70.94 lat (msec): min=16, max=184, avg=59.38, stdev=70.97 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 25], 60.00th=[ 25], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8827, max= 8827, per=100.00%, avg=8827.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=8.33%, 1000=16.67% cpu : usr=0.00%, sys=17.44%, ctx=999, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30453: Tue Apr 16 13:15:11 2024 read: IOPS=0, BW=606KiB/s (620kB/s)(2048KiB/3380msec) clat (msec): min=45, max=1063, avg=554.40, stdev=719.46 lat (msec): min=45, max=1063, avg=554.40, stdev=719.46 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 1062], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 1718, max= 2048, per=31.08%, avg=1883.00, stdev=233.35, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=3030KiB/s (3102kB/s)(10.0MiB/3380msec) clat (msec): min=15, max=1030, avg=226.41, stdev=397.31 lat (msec): min=15, max=1030, avg=226.46, stdev=397.32 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 20], 40.00th=[ 28], 50.00th=[ 37], 60.00th=[ 45], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 927], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 6872, max=10240, per=100.00%, avg=8556.00, stdev=2381.54, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=16.67%, 1000=8.33% cpu : usr=0.03%, sys=12.52%, ctx=383, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30454: Tue Apr 16 13:15:11 2024 read: IOPS=1, BW=1832KiB/s (1876kB/s)(6144KiB/3353msec) clat (usec): min=346, max=1048.4k, avg=337053.02, stdev=505831.49 lat (usec): min=347, max=1048.4k, avg=337053.80, stdev=505831.27 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], | 20.00th=[ 379], 30.00th=[ 379], 40.00th=[ 441], | 50.00th=[ 441], 60.00th=[ 45351], 70.00th=[ 926942], | 80.00th=[ 926942], 90.00th=[1044382], 95.00th=[1044382], | 99.00th=[1044382], 99.50th=[1044382], 99.90th=[1044382], | 99.95th=[1044382], 99.99th=[1044382] bw ( KiB/s): min= 3430, max= 6144, per=79.01%, avg=4787.00, stdev=1919.09, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1832KiB/s (1876kB/s)(6144KiB/3353msec) clat (msec): min=17, max=1094, avg=220.48, stdev=429.47 lat (msec): min=17, max=1094, avg=220.53, stdev=429.46 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 74], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 3430, max= 4096, per=45.67%, avg=3874.00, stdev=384.52, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=10.08%, ctx=376, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30455: Tue Apr 16 13:15:11 2024 read: IOPS=2, BW=2236KiB/s (2289kB/s)(5120KiB/2290msec) clat (usec): min=647, max=883038, avg=205710.36, stdev=380652.61 lat (usec): min=648, max=883039, avg=205711.49, stdev=380652.34 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 652], | 30.00th=[ 775], 40.00th=[ 775], 50.00th=[ 49546], 60.00th=[ 49546], | 70.00th=[ 94897], 80.00th=[ 94897], 90.00th=[884999], 95.00th=[884999], | 99.00th=[884999], 99.50th=[884999], 99.90th=[884999], 99.95th=[884999], | 99.99th=[884999] bw ( KiB/s): min= 2048, max= 2048, per=33.80%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3130KiB/s (3205kB/s)(7168KiB/2290msec) clat (msec): min=22, max=1005, avg=178.62, stdev=364.84 lat (msec): min=22, max=1005, avg=178.66, stdev=364.84 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 41], 60.00th=[ 44], | 70.00th=[ 44], 80.00th=[ 77], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 2048, max= 6144, per=48.29%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 50=50.00%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=18.39%, ctx=370, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6059KiB/s (6205kB/s), 606KiB/s-2236KiB/s (620kB/s-2289kB/s), io=20.0MiB (20.0MB), run=2290-3380msec WRITE: bw=8483KiB/s (8686kB/s), 1550KiB/s-3130KiB/s (1587kB/s-3205kB/s), io=28.0MiB (29.4MB), run=2290-3380msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30456: Tue Apr 16 13:15:11 2024 read: IOPS=1, BW=2008KiB/s (2056kB/s)(7168KiB/3570msec) clat (msec): min=34, max=1059, avg=328.12, stdev=454.54 lat (msec): min=34, max=1059, avg=328.12, stdev=454.54 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 72], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 919], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 2048, max= 6144, per=55.19%, avg=3072.00, stdev=2048.00, samples=4 iops : min= 2, max= 6, avg= 3.00, stdev= 2.00, samples=4 write: IOPS=1, BW=1434KiB/s (1469kB/s)(5120KiB/3570msec) clat (msec): min=39, max=837, avg=253.34, stdev=338.54 lat (msec): min=39, max=838, avg=253.41, stdev=338.56 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2048, max= 4096, per=35.04%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 50=16.67%, 100=41.67%, 250=8.33%, 500=8.33%, 1000=16.67% cpu : usr=0.03%, sys=4.29%, ctx=96, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30457: Tue Apr 16 13:15:11 2024 read: IOPS=0, BW=557KiB/s (570kB/s)(2048KiB/3679msec) clat (usec): min=19403, max=57121, avg=38262.56, stdev=26670.29 lat (usec): min=19404, max=57122, avg=38263.36, stdev=26670.21 clat percentiles (usec): | 1.00th=[19530], 5.00th=[19530], 10.00th=[19530], 20.00th=[19530], | 30.00th=[19530], 40.00th=[19530], 50.00th=[19530], 60.00th=[56886], | 70.00th=[56886], 80.00th=[56886], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 2043, max= 2048, per=36.75%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2783KiB/s (2850kB/s)(10.0MiB/3679msec) clat (msec): min=46, max=1101, avg=359.77, stdev=464.89 lat (msec): min=46, max=1101, avg=359.84, stdev=464.87 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 68], 50.00th=[ 75], 60.00th=[ 93], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max= 8192, per=61.32%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 20=8.33%, 50=8.33%, 100=50.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=2.99%, ctx=47, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30458: Tue Apr 16 13:15:11 2024 read: IOPS=1, BW=1736KiB/s (1777kB/s)(6144KiB/3540msec) clat (msec): min=18, max=875, avg=168.67, stdev=346.71 lat (msec): min=18, max=875, avg=168.67, stdev=346.71 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2048, max= 6144, per=61.32%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1736KiB/s (1777kB/s)(6144KiB/3540msec) clat (msec): min=48, max=2128, avg=420.49, stdev=837.94 lat (msec): min=49, max=2128, avg=420.55, stdev=837.94 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 2048, max= 4096, per=43.80%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.03%, sys=3.59%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30459: Tue Apr 16 13:15:11 2024 read: IOPS=1, BW=1436KiB/s (1470kB/s)(5120KiB/3566msec) clat (usec): min=20515, max=74904, avg=40090.02, stdev=24414.31 lat (usec): min=20517, max=74905, avg=40091.06, stdev=24414.18 clat percentiles (usec): | 1.00th=[20579], 5.00th=[20579], 10.00th=[20579], 20.00th=[20579], | 30.00th=[22676], 40.00th=[22676], 50.00th=[25560], 60.00th=[25560], | 70.00th=[56886], 80.00th=[56886], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=2010KiB/s (2058kB/s)(7168KiB/3566msec) clat (msec): min=62, max=1059, avg=480.47, stdev=484.07 lat (msec): min=62, max=1059, avg=480.54, stdev=484.08 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 176], 60.00th=[ 944], | 70.00th=[ 944], 80.00th=[ 986], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 2048, max= 6144, per=39.42%, avg=3072.00, stdev=2048.00, samples=4 iops : min= 2, max= 6, avg= 3.00, stdev= 2.00, samples=4 lat (msec) : 50=25.00%, 100=41.67%, 250=8.33%, 1000=16.67% cpu : usr=0.00%, sys=3.93%, ctx=144, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5567KiB/s (5700kB/s), 557KiB/s-2008KiB/s (570kB/s-2056kB/s), io=20.0MiB (20.0MB), run=3540-3679msec WRITE: bw=7793KiB/s (7980kB/s), 1434KiB/s-2783KiB/s (1469kB/s-2850kB/s), io=28.0MiB (29.4MB), run=3540-3679msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30469: Tue Apr 16 13:15:17 2024 read: IOPS=0, BW=3322KiB/s (3402kB/s)(12.0MiB/3699msec) clat (msec): min=185, max=2500, avg=1232.73, stdev=1173.41 lat (msec): min=185, max=2500, avg=1232.73, stdev=1173.41 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 1011], 50.00th=[ 1011], 60.00th=[ 1011], | 70.00th=[ 2500], 80.00th=[ 2500], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=42.37%, ctx=1028, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30470: Tue Apr 16 13:15:17 2024 read: IOPS=0, BW=1324KiB/s (1356kB/s)(4096KiB/3094msec) clat (nsec): min=185818k, max=185818k, avg=185817579.00, stdev= 0.00 lat (nsec): min=185819k, max=185819k, avg=185819430.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2648KiB/s (2711kB/s)(8192KiB/3094msec) clat (msec): min=382, max=2524, avg=1453.35, stdev=1514.62 lat (msec): min=382, max=2524, avg=1453.50, stdev=1514.66 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 2534], | 70.00th=[ 2534], 80.00th=[ 2534], 90.00th=[ 2534], 95.00th=[ 2534], | 99.00th=[ 2534], 99.50th=[ 2534], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=9.47%, ctx=175, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30471: Tue Apr 16 13:15:17 2024 write: IOPS=0, BW=3045KiB/s (3118kB/s)(12.0MiB/4035msec) clat (msec): min=308, max=2635, avg=1339.46, stdev=1186.08 lat (msec): min=309, max=2636, avg=1339.74, stdev=1186.08 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 1070], 50.00th=[ 1070], 60.00th=[ 1070], | 70.00th=[ 2635], 80.00th=[ 2635], 90.00th=[ 2635], 95.00th=[ 2635], | 99.00th=[ 2635], 99.50th=[ 2635], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=8.28%, ctx=75, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30472: Tue Apr 16 13:15:17 2024 write: IOPS=0, BW=4029KiB/s (4126kB/s)(12.0MiB/3050msec) clat (msec): min=194, max=2431, avg=1015.17, stdev=1231.50 lat (msec): min=194, max=2431, avg=1015.44, stdev=1231.48 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 422], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.03%, sys=11.51%, ctx=261, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4429KiB/s (4536kB/s), 1324KiB/s-3322KiB/s (1356kB/s-3402kB/s), io=16.0MiB (16.8MB), run=3094-3699msec WRITE: bw=8121KiB/s (8316kB/s), 2648KiB/s-4029KiB/s (2711kB/s-4126kB/s), io=32.0MiB (33.6MB), run=3050-4035msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30473: Tue Apr 16 13:15:17 2024 read: IOPS=0, BW=2904KiB/s (2974kB/s)(12.0MiB/4231msec) clat (msec): min=119, max=2942, avg=1409.86, stdev=1426.75 lat (msec): min=119, max=2942, avg=1409.86, stdev=1426.74 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 1167], 50.00th=[ 1167], 60.00th=[ 1167], | 70.00th=[ 2937], 80.00th=[ 2937], 90.00th=[ 2937], 95.00th=[ 2937], | 99.00th=[ 2937], 99.50th=[ 2937], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=2.58%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30474: Tue Apr 16 13:15:17 2024 read: IOPS=0, BW=977KiB/s (1001kB/s)(4096KiB/4192msec) clat (nsec): min=2866.9M, max=2866.9M, avg=2866914797.00, stdev= 0.00 lat (nsec): min=2866.9M, max=2866.9M, avg=2866915729.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 2869], 5.00th=[ 2869], 10.00th=[ 2869], 20.00th=[ 2869], | 30.00th=[ 2869], 40.00th=[ 2869], 50.00th=[ 2869], 60.00th=[ 2869], | 70.00th=[ 2869], 80.00th=[ 2869], 90.00th=[ 2869], 95.00th=[ 2869], | 99.00th=[ 2869], 99.50th=[ 2869], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1954KiB/s (2001kB/s)(8192KiB/4192msec) clat (msec): min=317, max=1004, avg=661.04, stdev=485.79 lat (msec): min=317, max=1004, avg=661.45, stdev=485.74 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 1003], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.05%, sys=3.53%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30475: Tue Apr 16 13:15:17 2024 write: IOPS=0, BW=2934KiB/s (3005kB/s)(12.0MiB/4188msec) clat (msec): min=140, max=2056, avg=1392.39, stdev=1084.88 lat (msec): min=140, max=2057, avg=1392.73, stdev=1084.76 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 1989], 50.00th=[ 1989], 60.00th=[ 1989], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=2.91%, ctx=46, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30476: Tue Apr 16 13:15:17 2024 write: IOPS=0, BW=2995KiB/s (3067kB/s)(12.0MiB/4103msec) clat (msec): min=226, max=3277, avg=1363.38, stdev=1667.47 lat (msec): min=226, max=3277, avg=1363.60, stdev=1667.46 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 3272], 80.00th=[ 3272], 90.00th=[ 3272], 95.00th=[ 3272], | 99.00th=[ 3272], 99.50th=[ 3272], 99.90th=[ 3272], 99.95th=[ 3272], | 99.99th=[ 3272] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.88%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3872KiB/s (3965kB/s), 977KiB/s-2904KiB/s (1001kB/s-2974kB/s), io=16.0MiB (16.8MB), run=4192-4231msec WRITE: bw=7817KiB/s (8004kB/s), 1954KiB/s-2995KiB/s (2001kB/s-3067kB/s), io=32.0MiB (33.6MB), run=4103-4192msec