== sanity test 398b: DIO and buffer IO race ============== 07:11:32 (1713438692) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.23845 s, 40.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15161: Thu Apr 18 07:11:49 2024 read: IOPS=104, BW=417KiB/s (427kB/s)(5992KiB/14357msec) clat (usec): min=49, max=197067, avg=5423.83, stdev=5871.42 lat (usec): min=49, max=197068, avg=5424.75, stdev=5871.52 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 137], 10.00th=[ 2474], 20.00th=[ 2999], | 30.00th=[ 3523], 40.00th=[ 4047], 50.00th=[ 4686], 60.00th=[ 5538], | 70.00th=[ 6325], 80.00th=[ 7373], 90.00th=[ 8979], 95.00th=[ 11076], | 99.00th=[ 16319], 99.50th=[ 18482], 99.90th=[ 24773], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 232, max= 638, per=24.66%, avg=412.29, stdev=71.62, samples=28 iops : min= 58, max= 159, avg=102.89, stdev=17.82, samples=28 write: IOPS=109, BW=439KiB/s (449kB/s)(6296KiB/14357msec) clat (usec): min=415, max=26436, avg=3923.79, stdev=3531.00 lat (usec): min=416, max=26437, avg=3926.23, stdev=3530.10 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 586], 10.00th=[ 652], 20.00th=[ 824], | 30.00th=[ 1037], 40.00th=[ 2114], 50.00th=[ 3130], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8848], 95.00th=[10945], | 99.00th=[14746], 99.50th=[15795], 99.90th=[24773], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 288, max= 646, per=24.79%, avg=434.00, stdev=76.77, samples=28 iops : min= 72, max= 161, avg=108.32, stdev=19.20, samples=28 lat (usec) : 50=0.03%, 100=1.37%, 250=1.37%, 500=0.26%, 750=8.20% lat (usec) : 1000=6.09% lat (msec) : 2=6.15%, 4=26.76%, 10=42.71%, 20=6.77%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.35%, ctx=6150, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15162: Thu Apr 18 07:11:49 2024 read: IOPS=105, BW=421KiB/s (431kB/s)(5940KiB/14121msec) clat (usec): min=59, max=25813, avg=5145.41, stdev=2860.50 lat (usec): min=60, max=25814, avg=5146.16, stdev=2860.53 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 1942], 10.00th=[ 2474], 20.00th=[ 3064], | 30.00th=[ 3490], 40.00th=[ 3949], 50.00th=[ 4621], 60.00th=[ 5342], | 70.00th=[ 6128], 80.00th=[ 7046], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[15008], 99.50th=[16909], 99.90th=[21627], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 232, max= 536, per=25.09%, avg=419.54, stdev=63.50, samples=28 iops : min= 58, max= 134, avg=104.75, stdev=15.82, samples=28 write: IOPS=112, BW=450KiB/s (460kB/s)(6348KiB/14121msec) clat (usec): min=495, max=186630, avg=4059.97, stdev=5725.25 lat (usec): min=496, max=186631, avg=4062.21, stdev=5724.66 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 594], 10.00th=[ 668], 20.00th=[ 832], | 30.00th=[ 1057], 40.00th=[ 2409], 50.00th=[ 3261], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8848], 95.00th=[ 10683], | 99.00th=[ 14615], 99.50th=[ 17171], 99.90th=[ 25035], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 288, max= 592, per=25.48%, avg=446.07, stdev=71.49, samples=28 iops : min= 72, max= 148, avg=111.39, stdev=17.89, samples=28 lat (usec) : 100=1.11%, 250=1.11%, 500=0.07%, 750=8.07%, 1000=6.38% lat (msec) : 2=5.24%, 4=29.04%, 10=42.90%, 20=5.83%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.87%, ctx=6061, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15163: Thu Apr 18 07:11:49 2024 read: IOPS=105, BW=422KiB/s (432kB/s)(6028KiB/14275msec) clat (usec): min=50, max=27520, avg=5212.29, stdev=2798.21 lat (usec): min=50, max=27520, avg=5215.35, stdev=2800.03 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 652], 10.00th=[ 2540], 20.00th=[ 3032], | 30.00th=[ 3523], 40.00th=[ 4080], 50.00th=[ 4817], 60.00th=[ 5538], | 70.00th=[ 6325], 80.00th=[ 7242], 90.00th=[ 8717], 95.00th=[10028], | 99.00th=[14222], 99.50th=[15664], 99.90th=[19792], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 232, max= 608, per=25.05%, avg=418.79, stdev=71.91, samples=28 iops : min= 58, max= 152, avg=104.64, stdev=17.97, samples=28 write: IOPS=109, BW=439KiB/s (449kB/s)(6260KiB/14275msec) clat (usec): min=490, max=201705, avg=4066.30, stdev=6111.29 lat (usec): min=490, max=201706, avg=4067.35, stdev=6111.19 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 594], 10.00th=[ 660], 20.00th=[ 840], | 30.00th=[ 1029], 40.00th=[ 2278], 50.00th=[ 3130], 60.00th=[ 3916], | 70.00th=[ 5014], 80.00th=[ 6587], 90.00th=[ 8717], 95.00th=[ 10945], | 99.00th=[ 15270], 99.50th=[ 18220], 99.90th=[ 22676], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 312, max= 600, per=24.72%, avg=432.79, stdev=69.59, samples=28 iops : min= 78, max= 150, avg=108.14, stdev=17.41, samples=28 lat (usec) : 100=1.53%, 250=0.88%, 500=0.10%, 750=7.68%, 1000=7.10% lat (msec) : 2=5.14%, 4=27.38%, 10=44.30%, 20=5.73%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.60%, ctx=6177, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15164: Thu Apr 18 07:11:49 2024 read: IOPS=105, BW=422KiB/s (432kB/s)(6052KiB/14338msec) clat (usec): min=49, max=31967, avg=5288.21, stdev=2955.99 lat (usec): min=49, max=31967, avg=5289.32, stdev=2955.82 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 2089], 10.00th=[ 2507], 20.00th=[ 3032], | 30.00th=[ 3523], 40.00th=[ 4113], 50.00th=[ 4752], 60.00th=[ 5538], | 70.00th=[ 6390], 80.00th=[ 7308], 90.00th=[ 8979], 95.00th=[10159], | 99.00th=[14877], 99.50th=[17171], 99.90th=[23200], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 224, max= 568, per=24.96%, avg=417.25, stdev=64.70, samples=28 iops : min= 56, max= 142, avg=104.18, stdev=16.23, samples=28 write: IOPS=108, BW=435KiB/s (445kB/s)(6236KiB/14338msec) clat (usec): min=466, max=200860, avg=4041.34, stdev=6147.64 lat (usec): min=467, max=200861, avg=4042.19, stdev=6147.67 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 594], 10.00th=[ 644], 20.00th=[ 799], | 30.00th=[ 971], 40.00th=[ 1860], 50.00th=[ 3195], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 6587], 90.00th=[ 8717], 95.00th=[ 10945], | 99.00th=[ 16057], 99.50th=[ 19792], 99.90th=[ 26608], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 256, max= 656, per=24.56%, avg=430.04, stdev=82.11, samples=28 iops : min= 64, max= 164, avg=107.39, stdev=20.54, samples=28 lat (usec) : 50=0.03%, 100=1.17%, 250=0.88%, 500=0.10%, 750=8.27% lat (usec) : 1000=7.32% lat (msec) : 2=4.88%, 4=27.21%, 10=43.78%, 20=6.05%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.47%, ctx=6229, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1672KiB/s (1713kB/s), 417KiB/s-422KiB/s (427kB/s-432kB/s), io=23.4MiB (24.6MB), run=14121-14357msec WRITE: bw=1751KiB/s (1793kB/s), 435KiB/s-450KiB/s (445kB/s-460kB/s), io=24.6MiB (25.7MB), run=14121-14357msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15157: Thu Apr 18 07:11:53 2024 read: IOPS=80, BW=323KiB/s (331kB/s)(5992KiB/18538msec) clat (usec): min=1642, max=220406, avg=5073.52, stdev=6829.37 lat (usec): min=1643, max=220407, avg=5074.26, stdev=6829.38 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 4146], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 10421], 95.00th=[ 13435], | 99.00th=[ 21103], 99.50th=[ 23987], 99.90th=[ 30802], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 152, max= 744, per=24.92%, avg=322.76, stdev=156.13, samples=37 iops : min= 38, max= 186, avg=80.65, stdev=39.05, samples=37 write: IOPS=84, BW=340KiB/s (348kB/s)(6296KiB/18538msec) clat (usec): min=2393, max=50554, avg=6912.57, stdev=4463.28 lat (usec): min=2393, max=50555, avg=6913.54, stdev=4463.27 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3032], 10.00th=[ 3261], 20.00th=[ 3556], | 30.00th=[ 3916], 40.00th=[ 4490], 50.00th=[ 5276], 60.00th=[ 6259], | 70.00th=[ 7832], 80.00th=[10028], 90.00th=[13042], 95.00th=[15664], | 99.00th=[22676], 99.50th=[26608], 99.90th=[30540], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 160, max= 712, per=24.97%, avg=338.54, stdev=156.26, samples=37 iops : min= 40, max= 178, avg=84.59, stdev=39.08, samples=37 lat (msec) : 2=2.77%, 4=42.25%, 10=39.55%, 20=14.00%, 50=1.37% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=11.23%, ctx=3686, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15158: Thu Apr 18 07:11:53 2024 read: IOPS=80, BW=322KiB/s (329kB/s)(5940KiB/18475msec) clat (usec): min=1654, max=196603, avg=5024.47, stdev=6272.71 lat (usec): min=1654, max=196605, avg=5025.23, stdev=6272.78 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2966], 50.00th=[ 3556], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6652], 90.00th=[ 10159], 95.00th=[ 13173], | 99.00th=[ 20317], 99.50th=[ 22676], 99.90th=[ 31589], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 80, max= 704, per=24.65%, avg=319.28, stdev=161.59, samples=36 iops : min= 20, max= 176, avg=79.78, stdev=40.42, samples=36 write: IOPS=85, BW=344KiB/s (352kB/s)(6348KiB/18475msec) clat (usec): min=2320, max=34579, avg=6904.52, stdev=4223.98 lat (usec): min=2321, max=34580, avg=6905.45, stdev=4224.04 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3032], 10.00th=[ 3261], 20.00th=[ 3589], | 30.00th=[ 3916], 40.00th=[ 4490], 50.00th=[ 5407], 60.00th=[ 6390], | 70.00th=[ 8094], 80.00th=[ 9896], 90.00th=[12911], 95.00th=[15139], | 99.00th=[21365], 99.50th=[25297], 99.90th=[30278], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 152, max= 752, per=24.91%, avg=337.72, stdev=149.78, samples=36 iops : min= 38, max= 188, avg=84.39, stdev=37.47, samples=36 lat (msec) : 2=2.77%, 4=41.86%, 10=40.04%, 20=14.16%, 50=1.14% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.24%, ctx=3691, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15159: Thu Apr 18 07:11:53 2024 read: IOPS=81, BW=327KiB/s (335kB/s)(6028KiB/18440msec) clat (usec): min=1636, max=30587, avg=4981.09, stdev=3798.54 lat (usec): min=1637, max=30587, avg=4981.84, stdev=3798.52 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2671], 40.00th=[ 3032], 50.00th=[ 3589], 60.00th=[ 4228], | 70.00th=[ 4948], 80.00th=[ 6783], 90.00th=[10552], 95.00th=[13173], | 99.00th=[18744], 99.50th=[21365], 99.90th=[26870], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 112, max= 784, per=24.98%, avg=323.50, stdev=150.19, samples=36 iops : min= 28, max= 196, avg=80.83, stdev=37.56, samples=36 write: IOPS=84, BW=339KiB/s (348kB/s)(6260KiB/18440msec) clat (msec): min=2, max=199, avg= 6.95, stdev= 6.50 lat (msec): min=2, max=199, avg= 6.95, stdev= 6.50 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 23], 99.50th=[ 27], 99.90th=[ 33], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 152, max= 784, per=24.68%, avg=334.61, stdev=157.70, samples=36 iops : min= 38, max= 196, avg=83.61, stdev=39.44, samples=36 lat (msec) : 2=2.64%, 4=41.60%, 10=40.79%, 20=13.64%, 50=1.30% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.35%, ctx=3629, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15160: Thu Apr 18 07:11:53 2024 read: IOPS=82, BW=328KiB/s (336kB/s)(6052KiB/18441msec) clat (usec): min=1634, max=31916, avg=4987.19, stdev=3916.29 lat (usec): min=1634, max=31917, avg=4987.91, stdev=3916.30 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1958], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2606], 40.00th=[ 2966], 50.00th=[ 3523], 60.00th=[ 4146], | 70.00th=[ 4883], 80.00th=[ 6718], 90.00th=[10945], 95.00th=[13566], | 99.00th=[19792], 99.50th=[21365], 99.90th=[26870], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 112, max= 720, per=24.98%, avg=323.50, stdev=150.77, samples=36 iops : min= 28, max= 180, avg=80.83, stdev=37.72, samples=36 write: IOPS=84, BW=338KiB/s (346kB/s)(6236KiB/18441msec) clat (msec): min=2, max=200, avg= 6.95, stdev= 6.45 lat (msec): min=2, max=200, avg= 6.95, stdev= 6.45 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 23], 99.50th=[ 27], 99.90th=[ 37], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 152, max= 720, per=24.61%, avg=333.72, stdev=151.81, samples=36 iops : min= 38, max= 180, avg=83.39, stdev=37.97, samples=36 lat (msec) : 2=3.09%, 4=41.21%, 10=40.40%, 20=14.00%, 50=1.27% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.39%, ctx=3590, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1295KiB/s (1326kB/s), 322KiB/s-328KiB/s (329kB/s-336kB/s), io=23.4MiB (24.6MB), run=18440-18538msec WRITE: bw=1356KiB/s (1389kB/s), 338KiB/s-344KiB/s (346kB/s-352kB/s), io=24.6MiB (25.7MB), run=18440-18538msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15195: Thu Apr 18 07:12:00 2024 read: IOPS=61, BW=985KiB/s (1009kB/s)(5920KiB/6010msec) clat (usec): min=1923, max=30787, avg=6392.31, stdev=4340.17 lat (usec): min=1924, max=30787, avg=6393.09, stdev=4340.16 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2507], 10.00th=[ 2835], 20.00th=[ 3163], | 30.00th=[ 3425], 40.00th=[ 3916], 50.00th=[ 4424], 60.00th=[ 5604], | 70.00th=[ 7177], 80.00th=[10290], 90.00th=[12387], 95.00th=[14615], | 99.00th=[21103], 99.50th=[23725], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 704, max= 1440, per=25.93%, avg=981.33, stdev=234.09, samples=12 iops : min= 44, max= 90, avg=61.33, stdev=14.63, samples=12 write: IOPS=66, BW=1060KiB/s (1085kB/s)(6368KiB/6010msec) clat (usec): min=2973, max=58837, avg=9136.95, stdev=5855.99 lat (usec): min=2975, max=58838, avg=9138.55, stdev=5855.96 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 3916], 10.00th=[ 4293], 20.00th=[ 4752], | 30.00th=[ 5276], 40.00th=[ 5997], 50.00th=[ 7177], 60.00th=[ 8586], | 70.00th=[10814], 80.00th=[12780], 90.00th=[16909], 95.00th=[20841], | 99.00th=[27657], 99.50th=[33424], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 864, max= 1184, per=26.59%, avg=1058.67, stdev=100.69, samples=12 iops : min= 54, max= 74, avg=66.17, stdev= 6.29, samples=12 lat (msec) : 2=0.13%, 4=22.79%, 10=50.65%, 20=22.27%, 50=4.04% lat (msec) : 100=0.13% cpu : usr=0.20%, sys=10.50%, ctx=911, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15196: Thu Apr 18 07:12:00 2024 read: IOPS=56, BW=908KiB/s (929kB/s)(5744KiB/6329msec) clat (usec): min=1912, max=28192, avg=6562.45, stdev=4492.92 lat (usec): min=1913, max=28192, avg=6563.16, stdev=4492.92 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2638], 10.00th=[ 2802], 20.00th=[ 3228], | 30.00th=[ 3556], 40.00th=[ 3851], 50.00th=[ 4621], 60.00th=[ 5866], | 70.00th=[ 7570], 80.00th=[ 9765], 90.00th=[13304], 95.00th=[16057], | 99.00th=[20841], 99.50th=[24249], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 640, max= 1152, per=23.74%, avg=898.25, stdev=173.99, samples=12 iops : min= 40, max= 72, avg=56.00, stdev=10.74, samples=12 write: IOPS=64, BW=1034KiB/s (1059kB/s)(6544KiB/6329msec) clat (usec): min=3057, max=65647, avg=9694.77, stdev=6352.20 lat (usec): min=3057, max=65649, avg=9696.25, stdev=6352.26 clat percentiles (usec): | 1.00th=[ 3392], 5.00th=[ 4047], 10.00th=[ 4293], 20.00th=[ 4948], | 30.00th=[ 5538], 40.00th=[ 6521], 50.00th=[ 7439], 60.00th=[ 9372], | 70.00th=[11076], 80.00th=[13566], 90.00th=[17695], 95.00th=[20841], | 99.00th=[30278], 99.50th=[33424], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 830, max= 1152, per=25.51%, avg=1015.58, stdev=94.90, samples=12 iops : min= 51, max= 72, avg=63.33, stdev= 6.07, samples=12 lat (msec) : 2=0.13%, 4=22.66%, 10=48.83%, 20=24.48%, 50=3.78% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=9.81%, ctx=896, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15197: Thu Apr 18 07:12:00 2024 read: IOPS=65, BW=1043KiB/s (1068kB/s)(6256KiB/5999msec) clat (usec): min=2146, max=26396, avg=6735.28, stdev=4639.58 lat (usec): min=2147, max=26397, avg=6736.05, stdev=4639.66 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2573], 10.00th=[ 2868], 20.00th=[ 3294], | 30.00th=[ 3556], 40.00th=[ 4047], 50.00th=[ 4621], 60.00th=[ 5932], | 70.00th=[ 7701], 80.00th=[10683], 90.00th=[13173], 95.00th=[16712], | 99.00th=[23462], 99.50th=[25822], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 800, max= 1280, per=27.44%, avg=1038.36, stdev=149.21, samples=11 iops : min= 50, max= 80, avg=64.82, stdev= 9.43, samples=11 write: IOPS=62, BW=1006KiB/s (1030kB/s)(6032KiB/5999msec) clat (usec): min=3429, max=42903, avg=8906.84, stdev=5367.26 lat (usec): min=3431, max=42905, avg=8908.23, stdev=5367.30 clat percentiles (usec): | 1.00th=[ 3621], 5.00th=[ 3982], 10.00th=[ 4228], 20.00th=[ 4686], | 30.00th=[ 5407], 40.00th=[ 6128], 50.00th=[ 7504], 60.00th=[ 8455], | 70.00th=[ 9634], 80.00th=[12125], 90.00th=[15926], 95.00th=[19792], | 99.00th=[27132], 99.50th=[29754], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 736, max= 1120, per=24.84%, avg=988.91, stdev=115.83, samples=11 iops : min= 46, max= 70, avg=61.73, stdev= 7.27, samples=11 lat (msec) : 4=22.79%, 10=51.82%, 20=22.27%, 50=3.12% cpu : usr=0.22%, sys=10.39%, ctx=888, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15198: Thu Apr 18 07:12:00 2024 read: IOPS=62, BW=1007KiB/s (1031kB/s)(6032KiB/5992msec) clat (usec): min=2052, max=48548, avg=6987.65, stdev=5015.92 lat (usec): min=2052, max=48549, avg=6988.34, stdev=5015.93 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2573], 10.00th=[ 2769], 20.00th=[ 3261], | 30.00th=[ 3752], 40.00th=[ 4293], 50.00th=[ 5080], 60.00th=[ 6652], | 70.00th=[ 8225], 80.00th=[10945], 90.00th=[13173], 95.00th=[16057], | 99.00th=[23462], 99.50th=[30278], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 480, max= 1280, per=26.37%, avg=997.82, stdev=222.54, samples=11 iops : min= 30, max= 80, avg=62.36, stdev=13.91, samples=11 write: IOPS=65, BW=1044KiB/s (1069kB/s)(6256KiB/5992msec) clat (usec): min=3046, max=35318, avg=8566.88, stdev=5476.37 lat (usec): min=3047, max=35319, avg=8568.26, stdev=5476.51 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3818], 10.00th=[ 4080], 20.00th=[ 4490], | 30.00th=[ 4948], 40.00th=[ 5473], 50.00th=[ 6456], 60.00th=[ 7635], | 70.00th=[ 9372], 80.00th=[12125], 90.00th=[16319], 95.00th=[20317], | 99.00th=[28967], 99.50th=[30016], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 864, max= 1248, per=26.31%, avg=1047.27, stdev=119.81, samples=11 iops : min= 54, max= 78, avg=65.45, stdev= 7.49, samples=11 lat (msec) : 4=22.01%, 10=53.39%, 20=20.96%, 50=3.65% cpu : usr=0.20%, sys=10.27%, ctx=903, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3784KiB/s (3875kB/s), 908KiB/s-1043KiB/s (929kB/s-1068kB/s), io=23.4MiB (24.5MB), run=5992-6329msec WRITE: bw=3982KiB/s (4077kB/s), 1006KiB/s-1060KiB/s (1030kB/s-1085kB/s), io=24.6MiB (25.8MB), run=5992-6329msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15200: Thu Apr 18 07:12:00 2024 read: IOPS=57, BW=915KiB/s (937kB/s)(5920KiB/6469msec) clat (usec): min=53, max=39204, avg=12350.83, stdev=5278.55 lat (usec): min=53, max=39205, avg=12351.55, stdev=5278.55 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 95], 10.00th=[ 7177], 20.00th=[ 9896], | 30.00th=[10945], 40.00th=[11863], 50.00th=[12911], 60.00th=[13698], | 70.00th=[14615], 80.00th=[15664], 90.00th=[17433], 95.00th=[19530], | 99.00th=[25035], 99.50th=[27919], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 736, max= 1056, per=23.88%, avg=869.00, stdev=100.08, samples=12 iops : min= 46, max= 66, avg=54.17, stdev= 6.25, samples=12 write: IOPS=61, BW=984KiB/s (1008kB/s)(6368KiB/6469msec) clat (usec): min=794, max=25970, avg=4750.46, stdev=4442.38 lat (usec): min=795, max=25972, avg=4752.55, stdev=4442.42 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 898], 10.00th=[ 996], 20.00th=[ 1303], | 30.00th=[ 2147], 40.00th=[ 2900], 50.00th=[ 3523], 60.00th=[ 4113], | 70.00th=[ 5145], 80.00th=[ 6849], 90.00th=[ 9765], 95.00th=[14484], | 99.00th=[22414], 99.50th=[25297], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 608, max= 1280, per=24.71%, avg=946.17, stdev=188.98, samples=12 iops : min= 38, max= 80, avg=59.00, stdev=11.73, samples=12 lat (usec) : 100=2.60%, 250=1.69%, 1000=5.34% lat (msec) : 2=9.90%, 4=14.45%, 10=23.57%, 20=39.32%, 50=3.12% cpu : usr=0.23%, sys=20.45%, ctx=2549, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15201: Thu Apr 18 07:12:00 2024 read: IOPS=55, BW=881KiB/s (902kB/s)(5744KiB/6518msec) clat (usec): min=53, max=31640, avg=12678.98, stdev=4765.06 lat (usec): min=53, max=31640, avg=12680.19, stdev=4764.84 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 119], 10.00th=[ 8225], 20.00th=[10290], | 30.00th=[11076], 40.00th=[11994], 50.00th=[13042], 60.00th=[13829], | 70.00th=[14746], 80.00th=[15926], 90.00th=[17695], 95.00th=[19530], | 99.00th=[22938], 99.50th=[26346], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 704, max= 1408, per=24.00%, avg=873.38, stdev=184.90, samples=13 iops : min= 44, max= 88, avg=54.38, stdev=11.55, samples=13 write: IOPS=62, BW=1004KiB/s (1028kB/s)(6544KiB/6518msec) clat (usec): min=706, max=42303, avg=4785.31, stdev=5080.59 lat (usec): min=706, max=42304, avg=4786.61, stdev=5080.58 clat percentiles (usec): | 1.00th=[ 807], 5.00th=[ 906], 10.00th=[ 1004], 20.00th=[ 1237], | 30.00th=[ 1713], 40.00th=[ 2802], 50.00th=[ 3326], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 7308], 90.00th=[10421], 95.00th=[13698], | 99.00th=[24511], 99.50th=[33162], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 638, max= 1696, per=26.15%, avg=1001.38, stdev=318.67, samples=13 iops : min= 39, max= 106, avg=62.38, stdev=20.11, samples=13 lat (usec) : 100=1.82%, 250=1.30%, 500=0.13%, 750=0.39%, 1000=4.69% lat (msec) : 2=11.98%, 4=15.49%, 10=19.92%, 20=41.15%, 50=3.12% cpu : usr=0.08%, sys=20.25%, ctx=2508, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15202: Thu Apr 18 07:12:00 2024 read: IOPS=59, BW=951KiB/s (973kB/s)(6256KiB/6581msec) clat (usec): min=69, max=32429, avg=12433.55, stdev=5085.06 lat (usec): min=70, max=32430, avg=12434.25, stdev=5085.07 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 98], 10.00th=[ 7308], 20.00th=[10159], | 30.00th=[11338], 40.00th=[12256], 50.00th=[13042], 60.00th=[13829], | 70.00th=[14746], 80.00th=[15664], 90.00th=[17695], 95.00th=[18744], | 99.00th=[23987], 99.50th=[28967], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 704, max= 1664, per=25.62%, avg=932.46, stdev=241.09, samples=13 iops : min= 44, max= 104, avg=58.08, stdev=15.18, samples=13 write: IOPS=57, BW=917KiB/s (939kB/s)(6032KiB/6581msec) clat (usec): min=707, max=38436, avg=4537.27, stdev=4560.74 lat (usec): min=708, max=38437, avg=4538.72, stdev=4560.75 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 865], 10.00th=[ 930], 20.00th=[ 1188], | 30.00th=[ 1483], 40.00th=[ 2704], 50.00th=[ 3326], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6587], 90.00th=[10683], 95.00th=[13566], | 99.00th=[21627], 99.50th=[22938], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 512, max= 1152, per=22.94%, avg=878.31, stdev=182.99, samples=13 iops : min= 32, max= 72, avg=54.69, stdev=11.36, samples=13 lat (usec) : 100=2.73%, 250=1.69%, 500=0.13%, 750=0.39%, 1000=6.12% lat (msec) : 2=10.55%, 4=13.80%, 10=18.36%, 20=43.75%, 50=2.47% cpu : usr=0.15%, sys=20.23%, ctx=2578, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15203: Thu Apr 18 07:12:00 2024 read: IOPS=57, BW=920KiB/s (942kB/s)(6032KiB/6555msec) clat (usec): min=52, max=28817, avg=12445.66, stdev=5148.85 lat (usec): min=53, max=28818, avg=12446.42, stdev=5148.90 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 102], 10.00th=[ 7767], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[11731], 50.00th=[12518], 60.00th=[13435], | 70.00th=[14615], 80.00th=[16188], 90.00th=[17957], 95.00th=[20579], | 99.00th=[25560], 99.50th=[28705], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 704, max= 1344, per=24.82%, avg=903.08, stdev=176.54, samples=13 iops : min= 44, max= 84, avg=56.31, stdev=11.14, samples=13 write: IOPS=59, BW=954KiB/s (977kB/s)(6256KiB/6555msec) clat (usec): min=738, max=37048, avg=4741.82, stdev=4627.90 lat (usec): min=739, max=37049, avg=4743.29, stdev=4628.26 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 914], 10.00th=[ 1012], 20.00th=[ 1352], | 30.00th=[ 2278], 40.00th=[ 2966], 50.00th=[ 3392], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[10552], 95.00th=[15664], | 99.00th=[20579], 99.50th=[31851], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 768, max= 1408, per=24.55%, avg=939.92, stdev=176.32, samples=13 iops : min= 48, max= 88, avg=58.62, stdev=10.97, samples=13 lat (usec) : 100=2.34%, 250=1.82%, 750=0.13%, 1000=4.43% lat (msec) : 2=10.16%, 4=16.41%, 10=20.70%, 20=40.89%, 50=3.12% cpu : usr=0.06%, sys=20.20%, ctx=2581, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3640KiB/s (3727kB/s), 881KiB/s-951KiB/s (902kB/s-973kB/s), io=23.4MiB (24.5MB), run=6469-6581msec WRITE: bw=3829KiB/s (3921kB/s), 917KiB/s-1004KiB/s (939kB/s-1028kB/s), io=24.6MiB (25.8MB), run=6469-6581msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15214: Thu Apr 18 07:12:03 2024 read: IOPS=3, BW=3540KiB/s (3625kB/s)(7168KiB/2025msec) clat (usec): min=587, max=669385, avg=266639.19, stdev=261631.28 lat (usec): min=589, max=669386, avg=266640.65, stdev=261631.15 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 31589], | 30.00th=[ 90702], 40.00th=[ 90702], 50.00th=[147850], 60.00th=[442500], | 70.00th=[442500], 80.00th=[484443], 90.00th=[666895], 95.00th=[666895], | 99.00th=[666895], 99.50th=[666895], 99.90th=[666895], 99.95th=[666895], | 99.99th=[666895] bw ( KiB/s): min= 1841, max= 4096, per=31.69%, avg=3204.33, stdev=1199.21, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 write: IOPS=2, BW=2528KiB/s (2589kB/s)(5120KiB/2025msec) clat (usec): min=21847, max=42823, avg=29667.07, stdev=8301.41 lat (usec): min=21886, max=42911, avg=29740.13, stdev=8313.91 clat percentiles (usec): | 1.00th=[21890], 5.00th=[21890], 10.00th=[21890], 20.00th=[21890], | 30.00th=[23987], 40.00th=[23987], 50.00th=[27657], 60.00th=[27657], | 70.00th=[32113], 80.00th=[32113], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 1841, max= 3676, per=19.48%, avg=2758.50, stdev=1297.54, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=8.33%, 250=8.33%, 500=16.67%, 750=8.33% cpu : usr=0.10%, sys=28.21%, ctx=843, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15215: Thu Apr 18 07:12:03 2024 read: IOPS=1, BW=1420KiB/s (1454kB/s)(2048KiB/1442msec) clat (usec): min=70482, max=93991, avg=82237.43, stdev=16623.29 lat (usec): min=70484, max=93992, avg=82238.48, stdev=16622.85 clat percentiles (usec): | 1.00th=[70779], 5.00th=[70779], 10.00th=[70779], 20.00th=[70779], | 30.00th=[70779], 40.00th=[70779], 50.00th=[70779], 60.00th=[93848], | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 1793, max= 2048, per=18.99%, avg=1920.50, stdev=180.31, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=7101KiB/s (7272kB/s)(10.0MiB/1442msec) clat (msec): min=33, max=627, avg=125.07, stdev=180.09 lat (msec): min=33, max=627, avg=125.15, stdev=180.10 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 37], | 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 58], 60.00th=[ 69], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 133], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 5380, max=10240, per=55.16%, avg=7810.00, stdev=3436.54, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=23.32%, ctx=141, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15216: Thu Apr 18 07:12:03 2024 read: IOPS=7, BW=7604KiB/s (7786kB/s)(6144KiB/808msec) clat (usec): min=434, max=218950, avg=57407.85, stdev=81461.18 lat (usec): min=435, max=218952, avg=57409.32, stdev=81461.29 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 14222], | 30.00th=[ 14222], 40.00th=[ 25035], 50.00th=[ 25035], 60.00th=[ 27132], | 70.00th=[ 58983], 80.00th=[ 58983], 90.00th=[219153], 95.00th=[219153], | 99.00th=[219153], 99.50th=[219153], 99.90th=[219153], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 8192, max= 8192, per=81.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7604KiB/s (7786kB/s)(6144KiB/808msec) clat (msec): min=28, max=122, avg=73.95, stdev=32.70 lat (msec): min=28, max=123, avg=74.01, stdev=32.70 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 73], 50.00th=[ 73], 60.00th=[ 85], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 8192, max= 8192, per=57.86%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=25.28%, ctx=229, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15218: Thu Apr 18 07:12:03 2024 read: IOPS=6, BW=6298KiB/s (6449kB/s)(5120KiB/813msec) clat (usec): min=369, max=242040, avg=74062.92, stdev=108660.71 lat (usec): min=369, max=242042, avg=74063.77, stdev=108661.22 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 371], | 30.00th=[ 570], 40.00th=[ 570], 50.00th=[ 570], 60.00th=[ 570], | 70.00th=[126354], 80.00th=[126354], 90.00th=[242222], 95.00th=[242222], | 99.00th=[242222], 99.50th=[242222], 99.90th=[242222], 99.95th=[242222], | 99.99th=[242222] bw ( KiB/s): min= 8192, max= 8192, per=81.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8817KiB/s (9028kB/s)(7168KiB/813msec) clat (msec): min=25, max=100, avg=59.78, stdev=24.44 lat (msec): min=25, max=100, avg=59.85, stdev=24.43 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 52], 60.00th=[ 67], | 70.00th=[ 67], 80.00th=[ 80], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min=12288, max=12288, per=86.79%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=25.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=30.42%, ctx=219, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.88MiB/s (10.4MB/s), 1420KiB/s-7604KiB/s (1454kB/s-7786kB/s), io=20.0MiB (20.0MB), run=808-2025msec WRITE: bw=13.8MiB/s (14.5MB/s), 2528KiB/s-8817KiB/s (2589kB/s-9028kB/s), io=28.0MiB (29.4MB), run=808-2025msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15217: Thu Apr 18 07:12:04 2024 read: IOPS=3, BW=3293KiB/s (3372kB/s)(7168KiB/2177msec) clat (msec): min=18, max=625, avg=208.43, stdev=234.75 lat (msec): min=18, max=625, avg=208.43, stdev=234.75 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 35], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 114], 60.00th=[ 146], | 70.00th=[ 146], 80.00th=[ 451], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 2048, max= 4096, per=36.29%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2352KiB/s (2408kB/s)(5120KiB/2177msec) clat (msec): min=47, max=478, avg=140.62, stdev=188.98 lat (msec): min=47, max=478, avg=140.68, stdev=188.99 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 2048, max= 3567, per=21.32%, avg=2807.50, stdev=1074.10, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=16.67%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=8.09%, ctx=77, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15219: Thu Apr 18 07:12:04 2024 read: IOPS=1, BW=1483KiB/s (1519kB/s)(2048KiB/1381msec) clat (usec): min=30605, max=36354, avg=33479.75, stdev=4064.89 lat (usec): min=30606, max=36355, avg=33481.15, stdev=4065.27 clat percentiles (usec): | 1.00th=[30540], 5.00th=[30540], 10.00th=[30540], 20.00th=[30540], | 30.00th=[30540], 40.00th=[30540], 50.00th=[30540], 60.00th=[36439], | 70.00th=[36439], 80.00th=[36439], 90.00th=[36439], 95.00th=[36439], | 99.00th=[36439], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 4096, max= 4096, per=43.54%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7415KiB/s (7593kB/s)(10.0MiB/1381msec) clat (msec): min=29, max=628, avg=129.95, stdev=181.43 lat (msec): min=30, max=628, avg=130.03, stdev=181.45 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 34], | 30.00th=[ 39], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 180], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 6144, max=10240, per=62.20%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=41.67%, 100=33.33%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=11.81%, ctx=88, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15220: Thu Apr 18 07:12:04 2024 read: IOPS=4, BW=4531KiB/s (4640kB/s)(6144KiB/1356msec) clat (msec): min=21, max=634, avg=161.55, stdev=239.62 lat (msec): min=21, max=634, avg=161.55, stdev=239.62 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 89], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 2048, max= 8192, per=54.43%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=4, BW=4531KiB/s (4640kB/s)(6144KiB/1356msec) clat (usec): min=36361, max=88627, avg=63768.70, stdev=18520.31 lat (usec): min=36432, max=88695, avg=63840.71, stdev=18523.99 clat percentiles (usec): | 1.00th=[36439], 5.00th=[36439], 10.00th=[36439], 20.00th=[48497], | 30.00th=[48497], 40.00th=[67634], 50.00th=[67634], 60.00th=[68682], | 70.00th=[72877], 80.00th=[72877], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 2048, max=10240, per=46.65%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 lat (msec) : 50=41.67%, 100=41.67%, 250=8.33%, 750=8.33% cpu : usr=0.07%, sys=12.76%, ctx=88, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15221: Thu Apr 18 07:12:04 2024 read: IOPS=3, BW=3568KiB/s (3654kB/s)(5120KiB/1435msec) clat (msec): min=20, max=679, avg=167.33, stdev=287.37 lat (msec): min=20, max=679, avg=167.33, stdev=287.37 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 36], 60.00th=[ 36], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 4071, max= 4096, per=43.41%, avg=4083.50, stdev=17.68, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4995KiB/s (5115kB/s)(7168KiB/1435msec) clat (msec): min=33, max=142, avg=83.21, stdev=37.95 lat (msec): min=33, max=142, avg=83.28, stdev=37.94 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 56], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 74], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 121], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 6144, max= 8143, per=54.24%, avg=7143.50, stdev=1413.51, samples=2 iops : min= 6, max= 7, avg= 6.50, stdev= 0.71, samples=2 lat (msec) : 50=33.33%, 100=41.67%, 250=16.67%, 750=8.33% cpu : usr=0.07%, sys=14.16%, ctx=132, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9407KiB/s (9633kB/s), 1483KiB/s-4531KiB/s (1519kB/s-4640kB/s), io=20.0MiB (20.0MB), run=1356-2177msec WRITE: bw=12.9MiB/s (13.5MB/s), 2352KiB/s-7415KiB/s (2408kB/s-7593kB/s), io=28.0MiB (29.4MB), run=1356-2177msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15228: Thu Apr 18 07:12:07 2024 read: IOPS=1, BW=7006KiB/s (7174kB/s)(12.0MiB/1754msec) clat (msec): min=8, max=1633, avg=584.15, stdev=910.19 lat (msec): min=8, max=1633, avg=584.15, stdev=910.19 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 10=33.33%, 250=33.33% cpu : usr=0.00%, sys=48.77%, ctx=751, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15229: Thu Apr 18 07:12:07 2024 read: IOPS=0, BW=1621KiB/s (1660kB/s)(4096KiB/2527msec) clat (nsec): min=135051k, max=135051k, avg=135051273.00, stdev= 0.00 lat (nsec): min=135053k, max=135053k, avg=135053062.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3242KiB/s (3320kB/s)(8192KiB/2527msec) clat (msec): min=735, max=1651, avg=1193.20, stdev=647.43 lat (msec): min=735, max=1651, avg=1193.50, stdev=647.46 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 735], 10.00th=[ 735], 20.00th=[ 735], | 30.00th=[ 735], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 1653], | 70.00th=[ 1653], 80.00th=[ 1653], 90.00th=[ 1653], 95.00th=[ 1653], | 99.00th=[ 1653], 99.50th=[ 1653], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 8192, max= 8192, per=63.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.04%, sys=10.29%, ctx=72, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15230: Thu Apr 18 07:12:07 2024 write: IOPS=1, BW=5088KiB/s (5210kB/s)(12.0MiB/2415msec) clat (msec): min=189, max=2029, avg=804.34, stdev=1060.76 lat (msec): min=189, max=2029, avg=804.57, stdev=1060.79 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 2022], 80.00th=[ 2022], 90.00th=[ 2022], 95.00th=[ 2022], | 99.00th=[ 2022], 99.50th=[ 2022], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 8192, max= 8192, per=63.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=14.08%, ctx=32, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15231: Thu Apr 18 07:12:07 2024 write: IOPS=1, BW=5378KiB/s (5507kB/s)(12.0MiB/2285msec) clat (msec): min=165, max=1620, avg=761.05, stdev=762.40 lat (msec): min=165, max=1620, avg=761.28, stdev=762.51 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 498], 50.00th=[ 498], 60.00th=[ 498], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8192, max= 8192, per=63.18%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=16.24%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6484KiB/s (6639kB/s), 1621KiB/s-7006KiB/s (1660kB/s-7174kB/s), io=16.0MiB (16.8MB), run=1754-2527msec WRITE: bw=12.7MiB/s (13.3MB/s), 3242KiB/s-5378KiB/s (3320kB/s-5507kB/s), io=32.0MiB (33.6MB), run=2285-2527msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15233: Thu Apr 18 07:12:07 2024 read: IOPS=1, BW=4546KiB/s (4655kB/s)(12.0MiB/2703msec) clat (msec): min=52, max=2088, avg=896.94, stdev=1061.47 lat (msec): min=52, max=2088, avg=896.94, stdev=1061.47 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2089], 95.00th=[ 2089], | 99.00th=[ 2089], 99.50th=[ 2089], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7529, max= 7529, per=100.00%, avg=7529.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.37%, ctx=49, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15234: Thu Apr 18 07:12:07 2024 read: IOPS=0, BW=1529KiB/s (1566kB/s)(4096KiB/2679msec) clat (nsec): min=347227k, max=347227k, avg=347226549.00, stdev= 0.00 lat (nsec): min=347228k, max=347228k, avg=347227754.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3058KiB/s (3131kB/s)(8192KiB/2679msec) clat (msec): min=584, max=1734, avg=1159.43, stdev=813.16 lat (msec): min=584, max=1734, avg=1159.69, stdev=813.17 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 1737], | 70.00th=[ 1737], 80.00th=[ 1737], 90.00th=[ 1737], 95.00th=[ 1737], | 99.00th=[ 1737], 99.50th=[ 1737], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7515, max= 7515, per=61.44%, avg=7515.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=6.31%, ctx=50, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15235: Thu Apr 18 07:12:07 2024 write: IOPS=1, BW=4754KiB/s (4868kB/s)(12.0MiB/2585msec) clat (msec): min=255, max=2013, avg=861.02, stdev=998.75 lat (msec): min=255, max=2014, avg=861.26, stdev=998.77 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 2022], 80.00th=[ 2022], 90.00th=[ 2022], 95.00th=[ 2022], | 99.00th=[ 2022], 99.50th=[ 2022], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 8192, max= 8192, per=66.98%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=8.01%, ctx=66, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15236: Thu Apr 18 07:12:07 2024 write: IOPS=1, BW=4973KiB/s (5092kB/s)(12.0MiB/2471msec) clat (msec): min=110, max=2003, avg=821.61, stdev=1030.57 lat (msec): min=110, max=2003, avg=821.88, stdev=1030.57 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 2005], 80.00th=[ 2005], 90.00th=[ 2005], 95.00th=[ 2005], | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 8192, max= 8192, per=66.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.55%, ctx=134, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6061KiB/s (6207kB/s), 1529KiB/s-4546KiB/s (1566kB/s-4655kB/s), io=16.0MiB (16.8MB), run=2679-2703msec WRITE: bw=11.9MiB/s (12.5MB/s), 3058KiB/s-4973KiB/s (3131kB/s-5092kB/s), io=32.0MiB (33.6MB), run=2471-2679msec