== sanity test 398b: DIO and buffer IO race ============== 18:02:10 (1713477730) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.34317 s, 37.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15428: Thu Apr 18 18:02:27 2024 read: IOPS=109, BW=437KiB/s (447kB/s)(5992KiB/13721msec) clat (usec): min=61, max=200172, avg=5144.49, stdev=5764.68 lat (usec): min=61, max=200174, avg=5145.19, stdev=5764.69 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 1893], 10.00th=[ 2343], 20.00th=[ 2835], | 30.00th=[ 3326], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5211], | 70.00th=[ 6063], 80.00th=[ 6980], 90.00th=[ 8356], 95.00th=[ 10159], | 99.00th=[ 13566], 99.50th=[ 15533], 99.90th=[ 27132], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 248, max= 648, per=24.93%, avg=432.07, stdev=75.82, samples=27 iops : min= 62, max= 162, avg=107.85, stdev=18.91, samples=27 write: IOPS=114, BW=459KiB/s (470kB/s)(6296KiB/13721msec) clat (usec): min=428, max=19682, avg=3779.95, stdev=3241.84 lat (usec): min=429, max=19683, avg=3780.89, stdev=3241.87 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 619], 20.00th=[ 783], | 30.00th=[ 1057], 40.00th=[ 2343], 50.00th=[ 3032], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6259], 90.00th=[ 8586], 95.00th=[10290], | 99.00th=[13435], 99.50th=[15270], 99.90th=[17957], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 296, max= 688, per=25.03%, avg=454.00, stdev=77.48, samples=27 iops : min= 74, max= 172, avg=113.33, stdev=19.44, samples=27 lat (usec) : 100=1.37%, 250=0.78%, 500=0.33%, 750=9.44%, 1000=4.92% lat (msec) : 2=5.01%, 4=30.34%, 10=42.45%, 20=5.27%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.45%, ctx=6121, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15429: Thu Apr 18 18:02:27 2024 read: IOPS=107, BW=429KiB/s (439kB/s)(5940KiB/13853msec) clat (usec): min=49, max=21543, avg=4970.22, stdev=2730.53 lat (usec): min=50, max=21544, avg=4971.39, stdev=2731.03 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 2024], 10.00th=[ 2376], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3818], 50.00th=[ 4490], 60.00th=[ 5145], | 70.00th=[ 5932], 80.00th=[ 6915], 90.00th=[ 8225], 95.00th=[ 9634], | 99.00th=[14353], 99.50th=[16712], 99.90th=[19268], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 215, max= 544, per=24.44%, avg=423.48, stdev=66.04, samples=27 iops : min= 53, max= 136, avg=105.74, stdev=16.59, samples=27 write: IOPS=114, BW=458KiB/s (469kB/s)(6348KiB/13853msec) clat (usec): min=471, max=184798, avg=4040.41, stdev=5698.20 lat (usec): min=471, max=184798, avg=4041.27, stdev=5698.21 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 791], | 30.00th=[ 1156], 40.00th=[ 2507], 50.00th=[ 3163], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[ 10683], | 99.00th=[ 15270], 99.50th=[ 17433], 99.90th=[ 33424], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 287, max= 576, per=24.59%, avg=446.04, stdev=73.86, samples=27 iops : min= 71, max= 144, avg=111.37, stdev=18.56, samples=27 lat (usec) : 50=0.03%, 100=1.60%, 250=0.36%, 500=0.26%, 750=9.28% lat (usec) : 1000=4.56% lat (msec) : 2=4.56%, 4=31.54%, 10=42.42%, 20=5.31%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.00%, ctx=6157, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15430: Thu Apr 18 18:02:27 2024 read: IOPS=110, BW=441KiB/s (451kB/s)(6028KiB/13672msec) clat (usec): min=61, max=20663, avg=4955.30, stdev=2636.72 lat (usec): min=62, max=20664, avg=4956.27, stdev=2636.48 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 2040], 10.00th=[ 2474], 20.00th=[ 2966], | 30.00th=[ 3392], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8160], 95.00th=[ 9503], | 99.00th=[13829], 99.50th=[17171], 99.90th=[19530], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 240, max= 528, per=25.30%, avg=438.44, stdev=63.89, samples=27 iops : min= 60, max= 132, avg=109.56, stdev=15.95, samples=27 write: IOPS=114, BW=458KiB/s (469kB/s)(6260KiB/13672msec) clat (usec): min=428, max=198483, avg=3922.02, stdev=5949.80 lat (usec): min=429, max=198484, avg=3923.87, stdev=5950.64 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 562], 10.00th=[ 611], 20.00th=[ 758], | 30.00th=[ 1012], 40.00th=[ 2147], 50.00th=[ 3064], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6390], 90.00th=[ 8717], 95.00th=[ 10290], | 99.00th=[ 14746], 99.50th=[ 16450], 99.90th=[ 22676], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 288, max= 640, per=25.07%, avg=454.74, stdev=75.88, samples=27 iops : min= 72, max= 160, avg=113.63, stdev=18.99, samples=27 lat (usec) : 100=1.50%, 250=0.55%, 500=0.39%, 750=9.70%, 1000=5.14% lat (msec) : 2=5.11%, 4=29.56%, 10=43.52%, 20=4.43%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.47%, ctx=6092, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15431: Thu Apr 18 18:02:27 2024 read: IOPS=110, BW=443KiB/s (453kB/s)(6052KiB/13674msec) clat (usec): min=49, max=18463, avg=4946.25, stdev=2687.76 lat (usec): min=50, max=18463, avg=4947.08, stdev=2688.02 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 1991], 10.00th=[ 2409], 20.00th=[ 2868], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5932], 80.00th=[ 6915], 90.00th=[ 8356], 95.00th=[ 9634], | 99.00th=[14615], 99.50th=[16319], 99.90th=[17957], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 248, max= 568, per=25.42%, avg=440.52, stdev=67.00, samples=27 iops : min= 62, max= 142, avg=110.07, stdev=16.77, samples=27 write: IOPS=114, BW=456KiB/s (467kB/s)(6236KiB/13674msec) clat (usec): min=394, max=187477, avg=3929.40, stdev=5717.38 lat (usec): min=395, max=187478, avg=3930.81, stdev=5717.08 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 553], 10.00th=[ 603], 20.00th=[ 758], | 30.00th=[ 1004], 40.00th=[ 2212], 50.00th=[ 3032], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[ 10683], | 99.00th=[ 13698], 99.50th=[ 13960], 99.90th=[ 22152], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 320, max= 592, per=25.08%, avg=455.00, stdev=66.42, samples=27 iops : min= 80, max= 148, avg=113.70, stdev=16.64, samples=27 lat (usec) : 50=0.07%, 100=1.27%, 250=0.42%, 500=0.55%, 750=9.64% lat (usec) : 1000=5.14% lat (msec) : 2=5.11%, 4=30.73%, 10=41.86%, 20=5.11%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.42%, ctx=6062, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1733KiB/s (1775kB/s), 429KiB/s-443KiB/s (439kB/s-453kB/s), io=23.4MiB (24.6MB), run=13672-13853msec WRITE: bw=1815KiB/s (1858kB/s), 456KiB/s-459KiB/s (467kB/s-470kB/s), io=24.6MiB (25.7MB), run=13672-13853msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15432: Thu Apr 18 18:02:31 2024 read: IOPS=85, BW=340KiB/s (349kB/s)(5992KiB/17604msec) clat (usec): min=1604, max=226564, avg=4966.37, stdev=6813.66 lat (usec): min=1604, max=226565, avg=4967.05, stdev=6813.68 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1893], 10.00th=[ 2008], 20.00th=[ 2212], | 30.00th=[ 2507], 40.00th=[ 2900], 50.00th=[ 3392], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 6849], 90.00th=[ 10290], 95.00th=[ 12780], | 99.00th=[ 19268], 99.50th=[ 21365], 99.90th=[ 27132], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 152, max= 800, per=24.83%, avg=338.71, stdev=157.87, samples=35 iops : min= 38, max= 200, avg=84.66, stdev=39.48, samples=35 write: IOPS=89, BW=358KiB/s (366kB/s)(6296KiB/17604msec) clat (usec): min=2084, max=33413, avg=6432.07, stdev=3966.46 lat (usec): min=2085, max=33414, avg=6432.92, stdev=3966.49 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2769], 10.00th=[ 3032], 20.00th=[ 3392], | 30.00th=[ 3785], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 5932], | 70.00th=[ 7308], 80.00th=[ 8979], 90.00th=[12125], 95.00th=[13960], | 99.00th=[19792], 99.50th=[22676], 99.90th=[31327], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 176, max= 848, per=24.90%, avg=355.63, stdev=159.73, samples=35 iops : min= 44, max= 212, avg=88.89, stdev=39.95, samples=35 lat (msec) : 2=4.72%, 4=41.47%, 10=40.14%, 20=12.86%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.21%, ctx=3614, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15433: Thu Apr 18 18:02:31 2024 read: IOPS=85, BW=340KiB/s (348kB/s)(5940KiB/17459msec) clat (usec): min=1549, max=191901, avg=4768.62, stdev=5990.31 lat (usec): min=1550, max=191902, avg=4769.36, stdev=5990.35 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1876], 10.00th=[ 2008], 20.00th=[ 2212], | 30.00th=[ 2507], 40.00th=[ 2868], 50.00th=[ 3326], 60.00th=[ 3949], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 9503], 95.00th=[ 12649], | 99.00th=[ 18482], 99.50th=[ 20841], 99.90th=[ 23725], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 111, max= 776, per=24.78%, avg=338.00, stdev=165.83, samples=34 iops : min= 27, max= 194, avg=84.41, stdev=41.50, samples=34 write: IOPS=90, BW=364KiB/s (372kB/s)(6348KiB/17459msec) clat (usec): min=2125, max=36437, avg=6518.39, stdev=4072.34 lat (usec): min=2126, max=36438, avg=6519.24, stdev=4072.37 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2835], 10.00th=[ 3064], 20.00th=[ 3392], | 30.00th=[ 3851], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 6063], | 70.00th=[ 7701], 80.00th=[ 9372], 90.00th=[11600], 95.00th=[14091], | 99.00th=[21890], 99.50th=[25560], 99.90th=[33424], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 191, max= 776, per=25.05%, avg=357.76, stdev=151.01, samples=34 iops : min= 47, max= 194, avg=89.35, stdev=37.82, samples=34 lat (msec) : 2=4.82%, 4=41.70%, 10=40.43%, 20=12.14%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=11.23%, ctx=3575, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15434: Thu Apr 18 18:02:31 2024 read: IOPS=85, BW=344KiB/s (352kB/s)(6028KiB/17545msec) clat (usec): min=1505, max=32557, avg=4721.99, stdev=3593.90 lat (usec): min=1506, max=32557, avg=4722.69, stdev=3593.88 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1876], 10.00th=[ 1991], 20.00th=[ 2245], | 30.00th=[ 2540], 40.00th=[ 2868], 50.00th=[ 3392], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6521], 90.00th=[ 9896], 95.00th=[12518], | 99.00th=[18482], 99.50th=[20579], 99.90th=[24249], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 120, max= 776, per=25.15%, avg=343.03, stdev=156.67, samples=35 iops : min= 30, max= 194, avg=85.74, stdev=39.14, samples=35 write: IOPS=89, BW=357KiB/s (365kB/s)(6260KiB/17545msec) clat (msec): min=2, max=190, avg= 6.64, stdev= 6.14 lat (msec): min=2, max=190, avg= 6.64, stdev= 6.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 32], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 176, max= 792, per=24.93%, avg=356.06, stdev=157.40, samples=35 iops : min= 44, max= 198, avg=89.00, stdev=39.33, samples=35 lat (msec) : 2=5.11%, 4=41.73%, 10=39.84%, 20=12.60%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.38%, ctx=3661, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15435: Thu Apr 18 18:02:31 2024 read: IOPS=86, BW=345KiB/s (354kB/s)(6052KiB/17527msec) clat (usec): min=1616, max=32616, avg=4725.27, stdev=3528.08 lat (usec): min=1617, max=32617, avg=4726.01, stdev=3528.05 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2540], 40.00th=[ 2933], 50.00th=[ 3392], 60.00th=[ 3982], | 70.00th=[ 4948], 80.00th=[ 6587], 90.00th=[ 9896], 95.00th=[12387], | 99.00th=[17695], 99.50th=[19268], 99.90th=[27657], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 144, max= 800, per=25.30%, avg=345.03, stdev=154.83, samples=35 iops : min= 36, max= 200, avg=86.17, stdev=38.75, samples=35 write: IOPS=88, BW=356KiB/s (364kB/s)(6236KiB/17527msec) clat (msec): min=2, max=191, avg= 6.64, stdev= 6.17 lat (msec): min=2, max=191, avg= 6.64, stdev= 6.17 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 38], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 168, max= 752, per=24.87%, avg=355.09, stdev=154.54, samples=35 iops : min= 42, max= 188, avg=88.69, stdev=38.68, samples=35 lat (msec) : 2=4.43%, 4=42.19%, 10=39.68%, 20=12.89%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.17%, ctx=3665, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1364KiB/s (1397kB/s), 340KiB/s-345KiB/s (348kB/s-354kB/s), io=23.4MiB (24.6MB), run=17459-17604msec WRITE: bw=1428KiB/s (1462kB/s), 356KiB/s-364KiB/s (364kB/s-372kB/s), io=24.6MiB (25.7MB), run=17459-17604msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15461: Thu Apr 18 18:02:38 2024 read: IOPS=55, BW=884KiB/s (905kB/s)(5920KiB/6696msec) clat (usec): min=75, max=55838, avg=12017.98, stdev=4658.66 lat (usec): min=75, max=55840, avg=12018.63, stdev=4658.72 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 7504], 10.00th=[ 8455], 20.00th=[ 9372], | 30.00th=[10159], 40.00th=[10814], 50.00th=[11469], 60.00th=[12125], | 70.00th=[13042], 80.00th=[14222], 90.00th=[15533], 95.00th=[17171], | 99.00th=[30540], 99.50th=[44303], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 672, max= 1024, per=24.63%, avg=876.15, stdev=122.46, samples=13 iops : min= 42, max= 64, avg=54.69, stdev= 7.76, samples=13 write: IOPS=59, BW=951KiB/s (974kB/s)(6368KiB/6696msec) clat (usec): min=776, max=50874, avg=5631.09, stdev=6055.50 lat (usec): min=777, max=50875, avg=5632.83, stdev=6056.33 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 832], 10.00th=[ 955], 20.00th=[ 2008], | 30.00th=[ 2474], 40.00th=[ 2868], 50.00th=[ 3523], 60.00th=[ 4686], | 70.00th=[ 6063], 80.00th=[ 8094], 90.00th=[12780], 95.00th=[16909], | 99.00th=[28967], 99.50th=[46924], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 672, max= 1181, per=25.45%, avg=952.38, stdev=155.39, samples=13 iops : min= 42, max= 73, avg=59.46, stdev= 9.61, samples=13 lat (usec) : 100=0.65%, 250=0.26%, 1000=5.73% lat (msec) : 2=4.56%, 4=19.01%, 10=27.21%, 20=39.71%, 50=2.60% lat (msec) : 100=0.26% cpu : usr=0.13%, sys=19.39%, ctx=2769, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15462: Thu Apr 18 18:02:38 2024 read: IOPS=55, BW=883KiB/s (904kB/s)(5744KiB/6503msec) clat (usec): min=67, max=48772, avg=11927.95, stdev=4485.79 lat (usec): min=68, max=48773, avg=11928.60, stdev=4485.82 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 7111], 10.00th=[ 8291], 20.00th=[ 9372], | 30.00th=[10290], 40.00th=[11076], 50.00th=[11731], 60.00th=[12256], | 70.00th=[12911], 80.00th=[14222], 90.00th=[15533], 95.00th=[17957], | 99.00th=[27657], 99.50th=[41681], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 608, max= 960, per=24.84%, avg=883.69, stdev=92.87, samples=13 iops : min= 38, max= 60, avg=55.23, stdev= 5.80, samples=13 write: IOPS=62, BW=1006KiB/s (1030kB/s)(6544KiB/6503msec) clat (usec): min=736, max=84670, avg=5408.76, stdev=6827.86 lat (usec): min=738, max=84672, avg=5409.97, stdev=6827.88 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 824], 10.00th=[ 881], 20.00th=[ 1369], | 30.00th=[ 2343], 40.00th=[ 2802], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5473], 80.00th=[ 7177], 90.00th=[13173], 95.00th=[16057], | 99.00th=[30540], 99.50th=[38536], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 704, max= 1440, per=26.77%, avg=1001.85, stdev=228.30, samples=13 iops : min= 44, max= 90, avg=62.62, stdev=14.27, samples=13 lat (usec) : 100=0.39%, 250=0.78%, 750=0.13%, 1000=8.20% lat (msec) : 2=5.34%, 4=16.80%, 10=27.21%, 20=38.41%, 50=2.60% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=20.04%, ctx=2639, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15463: Thu Apr 18 18:02:38 2024 read: IOPS=58, BW=929KiB/s (951kB/s)(6256KiB/6733msec) clat (usec): min=63, max=32339, avg=11732.61, stdev=4382.19 lat (usec): min=64, max=32340, avg=11735.80, stdev=4390.37 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 6849], 10.00th=[ 8029], 20.00th=[ 9110], | 30.00th=[10028], 40.00th=[10683], 50.00th=[11207], 60.00th=[11994], | 70.00th=[13042], 80.00th=[14222], 90.00th=[16319], 95.00th=[18744], | 99.00th=[27657], 99.50th=[28443], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 608, max= 1248, per=25.87%, avg=920.31, stdev=160.14, samples=13 iops : min= 38, max= 78, avg=57.38, stdev=10.04, samples=13 write: IOPS=55, BW=896KiB/s (917kB/s)(6032KiB/6733msec) clat (usec): min=786, max=73437, avg=5663.17, stdev=7069.11 lat (usec): min=787, max=73437, avg=5666.07, stdev=7069.98 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 889], 10.00th=[ 996], 20.00th=[ 1418], | 30.00th=[ 2507], 40.00th=[ 2966], 50.00th=[ 3654], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 7832], 90.00th=[12911], 95.00th=[16450], | 99.00th=[35914], 99.50th=[63701], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 544, max= 1120, per=23.02%, avg=861.23, stdev=183.57, samples=13 iops : min= 34, max= 70, avg=53.69, stdev=11.48, samples=13 lat (usec) : 100=1.43%, 250=0.39%, 500=0.13%, 1000=4.95% lat (msec) : 2=6.25%, 4=15.36%, 10=29.04%, 20=39.19%, 50=2.99% lat (msec) : 100=0.26% cpu : usr=0.13%, sys=19.40%, ctx=2744, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15464: Thu Apr 18 18:02:38 2024 read: IOPS=57, BW=917KiB/s (939kB/s)(6032KiB/6581msec) clat (usec): min=53, max=49377, avg=12021.22, stdev=4643.11 lat (usec): min=54, max=49378, avg=12022.98, stdev=4642.38 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 7111], 10.00th=[ 8160], 20.00th=[ 9241], | 30.00th=[10028], 40.00th=[10814], 50.00th=[11469], 60.00th=[12256], | 70.00th=[13042], 80.00th=[14091], 90.00th=[16188], 95.00th=[19006], | 99.00th=[31851], 99.50th=[37487], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 608, max= 1152, per=25.60%, avg=910.62, stdev=143.49, samples=13 iops : min= 38, max= 72, avg=56.85, stdev= 8.98, samples=13 write: IOPS=59, BW=951KiB/s (973kB/s)(6256KiB/6581msec) clat (usec): min=719, max=78424, avg=5216.27, stdev=6727.99 lat (usec): min=720, max=78427, avg=5217.37, stdev=6728.03 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 824], 10.00th=[ 914], 20.00th=[ 1598], | 30.00th=[ 2311], 40.00th=[ 2671], 50.00th=[ 3032], 60.00th=[ 3916], | 70.00th=[ 5211], 80.00th=[ 6915], 90.00th=[11076], 95.00th=[15533], | 99.00th=[30016], 99.50th=[54789], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 672, max= 1280, per=25.45%, avg=952.38, stdev=136.98, samples=13 iops : min= 42, max= 80, avg=59.46, stdev= 8.53, samples=13 lat (usec) : 100=0.65%, 250=0.26%, 500=0.13%, 750=0.39%, 1000=5.73% lat (msec) : 2=6.12%, 4=18.62%, 10=27.99%, 20=36.85%, 50=2.99% lat (msec) : 100=0.26% cpu : usr=0.09%, sys=19.54%, ctx=2735, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3557KiB/s (3643kB/s), 883KiB/s-929KiB/s (904kB/s-951kB/s), io=23.4MiB (24.5MB), run=6503-6733msec WRITE: bw=3743KiB/s (3833kB/s), 896KiB/s-1006KiB/s (917kB/s-1030kB/s), io=24.6MiB (25.8MB), run=6503-6733msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15465: Thu Apr 18 18:02:38 2024 read: IOPS=55, BW=883KiB/s (904kB/s)(5920KiB/6704msec) clat (usec): min=2006, max=54987, avg=6756.51, stdev=5521.45 lat (usec): min=2007, max=54988, avg=6757.20, stdev=5521.44 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2343], 10.00th=[ 2442], 20.00th=[ 2737], | 30.00th=[ 3097], 40.00th=[ 3556], 50.00th=[ 4424], 60.00th=[ 6128], | 70.00th=[ 8356], 80.00th=[10552], 90.00th=[13304], 95.00th=[15795], | 99.00th=[26084], 99.50th=[31065], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 576, max= 1216, per=24.79%, avg=876.15, stdev=167.05, samples=13 iops : min= 36, max= 76, avg=54.69, stdev=10.44, samples=13 write: IOPS=59, BW=950KiB/s (973kB/s)(6368KiB/6704msec) clat (usec): min=2455, max=67105, avg=10538.30, stdev=7103.97 lat (usec): min=2457, max=67106, avg=10539.68, stdev=7104.04 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3621], 10.00th=[ 4015], 20.00th=[ 5211], | 30.00th=[ 6390], 40.00th=[ 7439], 50.00th=[ 8586], 60.00th=[ 9896], | 70.00th=[11600], 80.00th=[14746], 90.00th=[19268], 95.00th=[23200], | 99.00th=[36439], 99.50th=[46400], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 576, max= 1216, per=25.61%, avg=952.38, stdev=191.89, samples=13 iops : min= 36, max= 76, avg=59.46, stdev=11.95, samples=13 lat (msec) : 4=27.60%, 10=40.62%, 20=26.17%, 50=5.34%, 100=0.26% cpu : usr=0.07%, sys=9.74%, ctx=1202, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15466: Thu Apr 18 18:02:38 2024 read: IOPS=55, BW=882KiB/s (903kB/s)(5744KiB/6515msec) clat (usec): min=1850, max=59808, avg=7245.62, stdev=6340.91 lat (usec): min=1850, max=59809, avg=7246.29, stdev=6340.93 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2802], | 30.00th=[ 3195], 40.00th=[ 3851], 50.00th=[ 4686], 60.00th=[ 6521], | 70.00th=[ 8717], 80.00th=[11600], 90.00th=[14222], 95.00th=[17957], | 99.00th=[33817], 99.50th=[38536], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 544, max= 1120, per=24.99%, avg=883.54, stdev=146.54, samples=13 iops : min= 34, max= 70, avg=55.15, stdev= 9.24, samples=13 write: IOPS=62, BW=1004KiB/s (1029kB/s)(6544KiB/6515msec) clat (usec): min=2544, max=89674, avg=9544.84, stdev=6868.12 lat (usec): min=2545, max=89678, avg=9546.18, stdev=6868.20 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3589], 10.00th=[ 3851], 20.00th=[ 4555], | 30.00th=[ 5538], 40.00th=[ 6587], 50.00th=[ 7767], 60.00th=[ 9634], | 70.00th=[10814], 80.00th=[13698], 90.00th=[16909], 95.00th=[21103], | 99.00th=[28967], 99.50th=[36963], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 672, max= 1376, per=26.93%, avg=1001.69, stdev=226.62, samples=13 iops : min= 42, max= 86, avg=62.54, stdev=14.25, samples=13 lat (msec) : 2=0.26%, 4=26.17%, 10=42.19%, 20=26.69%, 50=4.43% lat (msec) : 100=0.26% cpu : usr=0.12%, sys=9.72%, ctx=1133, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15467: Thu Apr 18 18:02:38 2024 read: IOPS=57, BW=923KiB/s (946kB/s)(6256KiB/6775msec) clat (usec): min=1902, max=55732, avg=7483.60, stdev=6580.11 lat (usec): min=1903, max=55733, avg=7484.27, stdev=6580.11 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2769], | 30.00th=[ 3228], 40.00th=[ 3851], 50.00th=[ 5080], 60.00th=[ 7242], | 70.00th=[ 9110], 80.00th=[11338], 90.00th=[13960], 95.00th=[18220], | 99.00th=[35914], 99.50th=[40109], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 479, max= 1277, per=26.10%, avg=922.62, stdev=215.94, samples=13 iops : min= 29, max= 79, avg=57.46, stdev=13.55, samples=13 write: IOPS=55, BW=890KiB/s (912kB/s)(6032KiB/6775msec) clat (usec): min=2863, max=67046, avg=10185.98, stdev=7036.38 lat (usec): min=2864, max=67049, avg=10187.56, stdev=7036.30 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3556], 10.00th=[ 4080], 20.00th=[ 4621], | 30.00th=[ 5932], 40.00th=[ 7373], 50.00th=[ 8455], 60.00th=[ 9896], | 70.00th=[11338], 80.00th=[14222], 90.00th=[18482], 95.00th=[22938], | 99.00th=[32113], 99.50th=[54264], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 544, max= 1152, per=23.15%, avg=861.08, stdev=194.65, samples=13 iops : min= 34, max= 72, avg=53.62, stdev=12.28, samples=13 lat (msec) : 2=0.39%, 4=26.17%, 10=40.89%, 20=26.43%, 50=5.73% lat (msec) : 100=0.39% cpu : usr=0.07%, sys=9.60%, ctx=1187, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15468: Thu Apr 18 18:02:38 2024 read: IOPS=57, BW=916KiB/s (938kB/s)(6032KiB/6585msec) clat (usec): min=1894, max=34671, avg=6603.03, stdev=5398.20 lat (usec): min=1894, max=34672, avg=6603.70, stdev=5398.30 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2212], 10.00th=[ 2409], 20.00th=[ 2671], | 30.00th=[ 3032], 40.00th=[ 3425], 50.00th=[ 3982], 60.00th=[ 5342], | 70.00th=[ 8225], 80.00th=[10290], 90.00th=[13960], 95.00th=[17957], | 99.00th=[24249], 99.50th=[31065], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 576, max= 1312, per=25.69%, avg=908.31, stdev=217.64, samples=13 iops : min= 36, max= 82, avg=56.77, stdev=13.60, samples=13 write: IOPS=59, BW=950KiB/s (973kB/s)(6256KiB/6585msec) clat (usec): min=2520, max=81856, avg=10453.19, stdev=8599.73 lat (usec): min=2521, max=81858, avg=10454.85, stdev=8600.47 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3589], 10.00th=[ 4015], 20.00th=[ 4686], | 30.00th=[ 5866], 40.00th=[ 7242], 50.00th=[ 8717], 60.00th=[10028], | 70.00th=[11600], 80.00th=[14484], 90.00th=[17957], 95.00th=[20841], | 99.00th=[56361], 99.50th=[79168], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 672, max= 1120, per=25.42%, avg=945.23, stdev=112.09, samples=13 iops : min= 42, max= 70, avg=59.08, stdev= 7.01, samples=13 lat (msec) : 2=0.52%, 4=29.17%, 10=39.45%, 20=25.78%, 50=4.56% lat (msec) : 100=0.52% cpu : usr=0.17%, sys=9.55%, ctx=1154, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3535KiB/s (3620kB/s), 882KiB/s-923KiB/s (903kB/s-946kB/s), io=23.4MiB (24.5MB), run=6515-6775msec WRITE: bw=3720KiB/s (3809kB/s), 890KiB/s-1004KiB/s (912kB/s-1029kB/s), io=24.6MiB (25.8MB), run=6515-6775msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15478: Thu Apr 18 18:02:42 2024 read: IOPS=3, BW=3112KiB/s (3187kB/s)(7168KiB/2303msec) clat (usec): min=525, max=977749, avg=288917.16, stdev=405771.91 lat (usec): min=527, max=977752, avg=288918.63, stdev=405772.42 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 11469], | 30.00th=[ 23462], 40.00th=[ 23462], 50.00th=[ 39060], 60.00th=[217056], | 70.00th=[217056], 80.00th=[750781], 90.00th=[977273], 95.00th=[977273], | 99.00th=[977273], 99.50th=[977273], 99.90th=[977273], 99.95th=[977273], | 99.99th=[977273] bw ( KiB/s): min= 2043, max= 6144, per=46.04%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=2, BW=2223KiB/s (2277kB/s)(5120KiB/2303msec) clat (usec): min=31002, max=90482, avg=55888.43, stdev=25643.07 lat (usec): min=31015, max=90545, avg=55964.08, stdev=25648.43 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[34866], 40.00th=[34866], 50.00th=[49021], 60.00th=[49021], | 70.00th=[73925], 80.00th=[73925], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 2048, max= 2048, per=16.45%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=8.33%, 1000=16.67% cpu : usr=0.13%, sys=23.59%, ctx=669, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15479: Thu Apr 18 18:02:42 2024 read: IOPS=0, BW=899KiB/s (921kB/s)(2048KiB/2278msec) clat (msec): min=42, max=769, avg=406.19, stdev=514.16 lat (msec): min=42, max=769, avg=406.19, stdev=514.16 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2048, max= 2048, per=23.03%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4495KiB/s (4603kB/s)(10.0MiB/2278msec) clat (msec): min=18, max=817, avg=146.39, stdev=250.46 lat (msec): min=18, max=817, avg=146.45, stdev=250.45 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 29], 40.00th=[ 31], 50.00th=[ 33], 60.00th=[ 44], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 300], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 2048, max= 8192, per=38.39%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 500=8.33%, 1000=16.67% cpu : usr=0.00%, sys=16.78%, ctx=352, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15480: Thu Apr 18 18:02:42 2024 read: IOPS=2, BW=2775KiB/s (2842kB/s)(6144KiB/2214msec) clat (usec): min=441, max=951695, avg=170627.88, stdev=383171.72 lat (usec): min=442, max=951697, avg=170629.08, stdev=383171.98 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], 20.00th=[ 578], | 30.00th=[ 578], 40.00th=[ 668], 50.00th=[ 668], 60.00th=[ 18220], | 70.00th=[ 52167], 80.00th=[ 52167], 90.00th=[952108], 95.00th=[952108], | 99.00th=[952108], 99.50th=[952108], 99.90th=[952108], 99.95th=[952108], | 99.99th=[952108] bw ( KiB/s): min= 4087, max= 4087, per=45.96%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2775KiB/s (2842kB/s)(6144KiB/2214msec) clat (msec): min=29, max=648, avg=198.18, stdev=230.20 lat (msec): min=29, max=648, avg=198.22, stdev=230.18 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 90], 50.00th=[ 90], 60.00th=[ 157], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 2043, max= 6131, per=32.83%, avg=4087.00, stdev=2890.65, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 20=8.33%, 50=8.33%, 100=25.00%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=13.51%, ctx=347, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15481: Thu Apr 18 18:02:42 2024 read: IOPS=2, BW=2279KiB/s (2333kB/s)(5120KiB/2247msec) clat (usec): min=493, max=781392, avg=321931.10, stdev=417566.19 lat (usec): min=494, max=781393, avg=321932.37, stdev=417565.80 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 494], 10.00th=[ 494], 20.00th=[ 494], | 30.00th=[ 898], 40.00th=[ 898], 50.00th=[ 50594], 60.00th=[ 50594], | 70.00th=[775947], 80.00th=[775947], 90.00th=[784335], 95.00th=[784335], | 99.00th=[784335], 99.50th=[784335], 99.90th=[784335], 99.95th=[784335], | 99.99th=[784335] bw ( KiB/s): min= 2048, max= 2048, per=23.03%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3190KiB/s (3267kB/s)(7168KiB/2247msec) clat (msec): min=25, max=252, avg=90.73, stdev=75.97 lat (msec): min=25, max=252, avg=90.79, stdev=75.97 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 34], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 88], 60.00th=[ 89], | 70.00th=[ 89], 80.00th=[ 91], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 8192, max= 8192, per=65.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 50=16.67%, 100=41.67%, 500=8.33%, 1000=16.67% cpu : usr=0.00%, sys=17.41%, ctx=574, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8893KiB/s (9106kB/s), 899KiB/s-3112KiB/s (921kB/s-3187kB/s), io=20.0MiB (20.0MB), run=2214-2303msec WRITE: bw=12.2MiB/s (12.7MB/s), 2223KiB/s-4495KiB/s (2277kB/s-4603kB/s), io=28.0MiB (29.4MB), run=2214-2303msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15483: Thu Apr 18 18:02:42 2024 read: IOPS=2, BW=2977KiB/s (3048kB/s)(7168KiB/2408msec) clat (msec): min=16, max=905, avg=289.06, stdev=350.99 lat (msec): min=16, max=905, avg=289.06, stdev=350.99 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 190], 60.00th=[ 222], | 70.00th=[ 222], 80.00th=[ 651], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2048, max= 6144, per=48.32%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2126KiB/s (2177kB/s)(5120KiB/2408msec) clat (msec): min=23, max=208, avg=76.66, stdev=74.74 lat (msec): min=23, max=208, avg=76.73, stdev=74.74 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 2048, max= 2048, per=17.26%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=8.33%, 250=25.00%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=6.65%, ctx=49, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15484: Thu Apr 18 18:02:42 2024 read: IOPS=0, BW=885KiB/s (906kB/s)(2048KiB/2314msec) clat (msec): min=212, max=683, avg=448.13, stdev=332.57 lat (msec): min=212, max=683, avg=448.13, stdev=332.57 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2043, max= 2048, per=24.13%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4425KiB/s (4531kB/s)(10.0MiB/2314msec) clat (msec): min=32, max=805, avg=141.60, stdev=235.25 lat (msec): min=32, max=805, avg=141.67, stdev=235.24 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 55], 50.00th=[ 61], 60.00th=[ 66], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 129], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 4096, max= 6131, per=40.23%, avg=4774.33, stdev=1174.91, samples=3 iops : min= 4, max= 5, avg= 4.33, stdev= 0.58, samples=3 lat (msec) : 50=25.00%, 100=33.33%, 250=25.00%, 750=8.33%, 1000=8.33% cpu : usr=0.04%, sys=8.26%, ctx=155, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15485: Thu Apr 18 18:02:42 2024 read: IOPS=2, BW=2543KiB/s (2604kB/s)(6144KiB/2416msec) clat (msec): min=21, max=874, avg=195.95, stdev=335.95 lat (msec): min=21, max=874, avg=195.95, stdev=335.95 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 56], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2043, max= 2048, per=24.13%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2543KiB/s (2604kB/s)(6144KiB/2416msec) clat (msec): min=20, max=662, avg=205.49, stdev=236.72 lat (msec): min=20, max=662, avg=205.56, stdev=236.74 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 161], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 6144, per=34.52%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 50=41.67%, 100=8.33%, 250=33.33%, 750=8.33%, 1000=8.33% cpu : usr=0.12%, sys=5.67%, ctx=223, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15486: Thu Apr 18 18:02:42 2024 read: IOPS=2, BW=2157KiB/s (2208kB/s)(5120KiB/2374msec) clat (msec): min=18, max=844, avg=324.75, stdev=419.20 lat (msec): min=18, max=844, avg=324.75, stdev=419.20 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 24], 60.00th=[ 24], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2048, max= 2048, per=24.16%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=3019KiB/s (3092kB/s)(7168KiB/2374msec) clat (msec): min=32, max=190, avg=106.25, stdev=65.92 lat (msec): min=32, max=190, avg=106.31, stdev=65.93 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 92], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 8192, max= 8192, per=69.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=16.67%, 250=25.00%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=5.56%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8477KiB/s (8680kB/s), 885KiB/s-2977KiB/s (906kB/s-3048kB/s), io=20.0MiB (20.0MB), run=2314-2416msec WRITE: bw=11.6MiB/s (12.2MB/s), 2126KiB/s-4425KiB/s (2177kB/s-4531kB/s), io=28.0MiB (29.4MB), run=2314-2416msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15493: Thu Apr 18 18:02:45 2024 read: IOPS=2, BW=10.4MiB/s (10.9MB/s)(12.0MiB/1155msec) clat (msec): min=67, max=880, avg=384.64, stdev=434.90 lat (msec): min=67, max=880, avg=384.64, stdev=434.90 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 8175, max= 8192, per=96.05%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=70.97%, ctx=180, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15494: Thu Apr 18 18:02:45 2024 read: IOPS=0, BW=2130KiB/s (2181kB/s)(4096KiB/1923msec) clat (nsec): min=106048k, max=106048k, avg=106048142.00, stdev= 0.00 lat (nsec): min=106050k, max=106050k, avg=106050489.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=96.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4260KiB/s (4362kB/s)(8192KiB/1923msec) clat (msec): min=306, max=1503, avg=905.23, stdev=846.68 lat (msec): min=306, max=1504, avg=905.48, stdev=846.64 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 1502], | 70.00th=[ 1502], 80.00th=[ 1502], 90.00th=[ 1502], 95.00th=[ 1502], | 99.00th=[ 1502], 99.50th=[ 1502], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=16.60%, ctx=573, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15495: Thu Apr 18 18:02:45 2024 write: IOPS=1, BW=6664KiB/s (6824kB/s)(12.0MiB/1844msec) clat (msec): min=359, max=1071, avg=613.46, stdev=397.15 lat (msec): min=360, max=1071, avg=613.73, stdev=397.18 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8175, max= 8192, per=48.03%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.05%, sys=19.91%, ctx=36, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15496: Thu Apr 18 18:02:45 2024 write: IOPS=1, BW=6454KiB/s (6609kB/s)(12.0MiB/1904msec) clat (msec): min=122, max=1187, avg=634.08, stdev=533.89 lat (msec): min=122, max=1187, avg=634.38, stdev=533.77 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 592], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8192, max= 8192, per=48.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=18.18%, ctx=34, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8520KiB/s (8725kB/s), 2130KiB/s-10.4MiB/s (2181kB/s-10.9MB/s), io=16.0MiB (16.8MB), run=1155-1923msec WRITE: bw=16.6MiB/s (17.4MB/s), 4260KiB/s-6664KiB/s (4362kB/s-6824kB/s), io=32.0MiB (33.6MB), run=1844-1923msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15497: Thu Apr 18 18:02:45 2024 read: IOPS=1, BW=5894KiB/s (6035kB/s)(12.0MiB/2085msec) clat (msec): min=451, max=1030, avg=694.73, stdev=300.26 lat (msec): min=451, max=1030, avg=694.74, stdev=300.26 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 600], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7787, max= 8192, per=100.00%, avg=7989.50, stdev=286.38, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=4.99%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15498: Thu Apr 18 18:02:45 2024 read: IOPS=0, BW=1975KiB/s (2022kB/s)(4096KiB/2074msec) clat (nsec): min=564943k, max=564943k, avg=564943146.00, stdev= 0.00 lat (nsec): min=564945k, max=564945k, avg=564945056.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3950KiB/s (4045kB/s)(8192KiB/2074msec) clat (msec): min=63, max=1443, avg=753.49, stdev=975.30 lat (msec): min=64, max=1443, avg=753.86, stdev=975.28 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 1452], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 8047, max= 8047, per=51.33%, avg=8047.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.05%, sys=5.45%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15499: Thu Apr 18 18:02:45 2024 write: IOPS=1, BW=6203KiB/s (6352kB/s)(12.0MiB/1981msec) clat (msec): min=386, max=1062, avg=659.37, stdev=356.20 lat (msec): min=386, max=1062, avg=659.61, stdev=356.23 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 531], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 8192, max= 8192, per=52.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.21%, ctx=82, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15500: Thu Apr 18 18:02:45 2024 write: IOPS=1, BW=5879KiB/s (6021kB/s)(12.0MiB/2090msec) clat (msec): min=98, max=1671, avg=695.97, stdev=851.73 lat (msec): min=99, max=1671, avg=696.21, stdev=851.80 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 1670], 80.00th=[ 1670], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.95%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7858KiB/s (8047kB/s), 1975KiB/s-5894KiB/s (2022kB/s-6035kB/s), io=16.0MiB (16.8MB), run=2074-2085msec WRITE: bw=15.3MiB/s (16.1MB/s), 3950KiB/s-6203KiB/s (4045kB/s-6352kB/s), io=32.0MiB (33.6MB), run=1981-2090msec