== sanity test 398b: DIO and buffer IO race ============== 22:34:19 (1713494059) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.44162 s, 34.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31725: Thu Apr 18 22:34:34 2024 read: IOPS=119, BW=477KiB/s (489kB/s)(5992KiB/12557msec) clat (usec): min=45, max=184317, avg=3896.18, stdev=5660.53 lat (usec): min=45, max=184318, avg=3896.76, stdev=5660.55 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 69], 10.00th=[ 74], 20.00th=[ 120], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6063], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[ 13829], 99.50th=[ 15664], 99.90th=[ 39060], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 200, max= 694, per=24.79%, avg=474.04, stdev=110.90, samples=25 iops : min= 50, max= 173, avg=118.40, stdev=27.72, samples=25 write: IOPS=125, BW=501KiB/s (513kB/s)(6296KiB/12557msec) clat (usec): min=464, max=39382, avg=4198.38, stdev=3946.39 lat (usec): min=464, max=39383, avg=4200.40, stdev=3946.20 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 668], | 30.00th=[ 988], 40.00th=[ 2343], 50.00th=[ 3458], 60.00th=[ 4359], | 70.00th=[ 5407], 80.00th=[ 6849], 90.00th=[ 9110], 95.00th=[11600], | 99.00th=[17695], 99.50th=[21365], 99.90th=[31327], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 224, max= 728, per=24.84%, avg=497.36, stdev=124.42, samples=25 iops : min= 56, max= 182, avg=124.24, stdev=31.11, samples=25 lat (usec) : 50=0.36%, 100=8.85%, 250=0.85%, 500=0.91%, 750=11.95% lat (usec) : 1000=2.83% lat (msec) : 2=5.76%, 4=27.44%, 10=35.19%, 20=5.44%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=24.47%, ctx=5996, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31726: Thu Apr 18 22:34:34 2024 read: IOPS=118, BW=475KiB/s (487kB/s)(5940KiB/12496msec) clat (usec): min=46, max=32512, avg=3624.97, stdev=3115.38 lat (usec): min=47, max=32513, avg=3626.28, stdev=3115.63 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 70], 10.00th=[ 74], 20.00th=[ 117], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3654], | 70.00th=[ 4424], 80.00th=[ 5800], 90.00th=[ 7570], 95.00th=[ 8848], | 99.00th=[13042], 99.50th=[16450], 99.90th=[28705], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 240, max= 688, per=24.84%, avg=474.87, stdev=111.86, samples=24 iops : min= 60, max= 172, avg=118.67, stdev=27.93, samples=24 write: IOPS=127, BW=508KiB/s (520kB/s)(6348KiB/12496msec) clat (usec): min=462, max=181458, avg=4412.07, stdev=6082.95 lat (usec): min=463, max=181459, avg=4413.15, stdev=6083.04 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 676], | 30.00th=[ 955], 40.00th=[ 2180], 50.00th=[ 3458], 60.00th=[ 4359], | 70.00th=[ 5604], 80.00th=[ 7308], 90.00th=[ 9372], 95.00th=[ 12256], | 99.00th=[ 17433], 99.50th=[ 21890], 99.90th=[ 42206], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 288, max= 742, per=24.95%, avg=499.54, stdev=106.19, samples=24 iops : min= 72, max= 185, avg=124.83, stdev=26.51, samples=24 lat (usec) : 50=0.23%, 100=8.92%, 250=1.24%, 500=0.75%, 750=11.59% lat (usec) : 1000=3.91% lat (msec) : 2=5.37%, 4=28.35%, 10=33.82%, 20=5.31%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=24.60%, ctx=5974, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31727: Thu Apr 18 22:34:34 2024 read: IOPS=120, BW=481KiB/s (492kB/s)(6028KiB/12536msec) clat (usec): min=46, max=179606, avg=3803.71, stdev=5642.67 lat (usec): min=47, max=179607, avg=3804.39, stdev=5642.65 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 69], 10.00th=[ 73], 20.00th=[ 99], | 30.00th=[ 2212], 40.00th=[ 2606], 50.00th=[ 3064], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 5932], 90.00th=[ 7767], 95.00th=[ 9372], | 99.00th=[ 13435], 99.50th=[ 20841], 99.90th=[ 44303], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 191, max= 688, per=25.12%, avg=480.20, stdev=112.32, samples=25 iops : min= 47, max= 172, avg=120.00, stdev=28.15, samples=25 write: IOPS=124, BW=499KiB/s (511kB/s)(6260KiB/12536msec) clat (usec): min=459, max=44398, avg=4276.94, stdev=4181.49 lat (usec): min=460, max=44399, avg=4278.17, stdev=4181.68 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 635], | 30.00th=[ 930], 40.00th=[ 2245], 50.00th=[ 3523], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7177], 90.00th=[ 9503], 95.00th=[11469], | 99.00th=[17433], 99.50th=[19792], 99.90th=[38536], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 271, max= 728, per=24.88%, avg=498.12, stdev=112.85, samples=25 iops : min= 67, max= 182, avg=124.48, stdev=28.27, samples=25 lat (usec) : 50=0.72%, 100=9.18%, 250=1.46%, 500=0.78%, 750=12.14% lat (usec) : 1000=2.86% lat (msec) : 2=5.50%, 4=26.86%, 10=34.24%, 20=5.76%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=24.76%, ctx=5991, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31728: Thu Apr 18 22:34:34 2024 read: IOPS=121, BW=486KiB/s (498kB/s)(6052KiB/12446msec) clat (usec): min=48, max=190012, avg=3780.97, stdev=5716.20 lat (usec): min=48, max=190013, avg=3781.80, stdev=5716.26 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 69], 10.00th=[ 73], 20.00th=[ 100], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3720], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[ 12649], 99.50th=[ 16188], 99.90th=[ 29492], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 224, max= 696, per=25.01%, avg=478.25, stdev=96.48, samples=24 iops : min= 56, max= 174, avg=119.54, stdev=24.11, samples=24 write: IOPS=125, BW=501KiB/s (513kB/s)(6236KiB/12446msec) clat (usec): min=435, max=31753, avg=4244.04, stdev=4162.69 lat (usec): min=436, max=31753, avg=4244.73, stdev=4162.70 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 627], | 30.00th=[ 840], 40.00th=[ 1795], 50.00th=[ 3326], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 7177], 90.00th=[ 9896], 95.00th=[12387], | 99.00th=[18482], 99.50th=[20317], 99.90th=[27657], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 272, max= 792, per=24.72%, avg=494.92, stdev=107.44, samples=24 iops : min= 68, max= 198, avg=123.71, stdev=26.85, samples=24 lat (usec) : 50=0.33%, 100=9.60%, 250=1.11%, 500=0.81%, 750=13.09% lat (usec) : 1000=2.83% lat (msec) : 2=5.73%, 4=26.60%, 10=33.53%, 20=5.96%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=24.61%, ctx=5980, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1912KiB/s (1958kB/s), 475KiB/s-486KiB/s (487kB/s-498kB/s), io=23.4MiB (24.6MB), run=12446-12557msec WRITE: bw=2002KiB/s (2050kB/s), 499KiB/s-508KiB/s (511kB/s-520kB/s), io=24.6MiB (25.7MB), run=12446-12557msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31721: Thu Apr 18 22:35:02 2024 read: IOPS=37, BW=151KiB/s (154kB/s)(5992KiB/39803msec) clat (usec): min=1631, max=187239, avg=3603.68, stdev=5957.30 lat (usec): min=1631, max=187241, avg=3604.29, stdev=5957.33 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2089], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2474], | 70.00th=[ 2704], 80.00th=[ 3458], 90.00th=[ 7111], 95.00th=[ 9634], | 99.00th=[ 18220], 99.50th=[ 21890], 99.90th=[ 60556], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 40, max= 272, per=25.05%, avg=150.30, stdev=59.22, samples=79 iops : min= 10, max= 68, avg=37.52, stdev=14.83, samples=79 write: IOPS=39, BW=158KiB/s (162kB/s)(6296KiB/39803msec) clat (usec): min=11955, max=56188, avg=21787.08, stdev=7457.39 lat (usec): min=11955, max=56189, avg=21787.87, stdev=7457.36 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13304], 10.00th=[13960], 20.00th=[15270], | 30.00th=[17171], 40.00th=[19268], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25297], 90.00th=[31589], 95.00th=[37487], | 99.00th=[47449], 99.50th=[51119], 99.90th=[55313], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 48, max= 208, per=25.14%, avg=157.90, stdev=42.86, samples=79 iops : min= 12, max= 52, avg=39.42, stdev=10.71, samples=79 lat (msec) : 2=9.83%, 4=30.63%, 10=6.02%, 20=24.97%, 50=28.22% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.09%, sys=4.41%, ctx=3310, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31722: Thu Apr 18 22:35:02 2024 read: IOPS=37, BW=148KiB/s (152kB/s)(5940KiB/40020msec) clat (usec): min=1647, max=181799, avg=3463.64, stdev=5558.74 lat (usec): min=1647, max=181801, avg=3464.28, stdev=5558.75 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2073], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2474], | 70.00th=[ 2704], 80.00th=[ 3294], 90.00th=[ 6521], 95.00th=[ 9372], | 99.00th=[ 16188], 99.50th=[ 23200], 99.90th=[ 38536], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 15, max= 280, per=24.71%, avg=148.28, stdev=61.60, samples=80 iops : min= 3, max= 70, avg=36.98, stdev=15.45, samples=80 write: IOPS=39, BW=159KiB/s (162kB/s)(6348KiB/40020msec) clat (usec): min=11877, max=67406, avg=21905.64, stdev=7482.10 lat (usec): min=11878, max=67407, avg=21906.41, stdev=7482.14 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13435], 10.00th=[14091], 20.00th=[15401], | 30.00th=[17433], 40.00th=[19530], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23725], 80.00th=[25297], 90.00th=[31851], 95.00th=[36963], | 99.00th=[49021], 99.50th=[51643], 99.90th=[57410], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 55, max= 200, per=25.22%, avg=158.37, stdev=42.06, samples=80 iops : min= 13, max= 50, avg=39.50, stdev=10.60, samples=80 lat (msec) : 2=10.16%, 4=30.53%, 10=5.73%, 20=23.37%, 50=29.79% lat (msec) : 100=0.39%, 250=0.03% cpu : usr=0.08%, sys=4.42%, ctx=3330, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31723: Thu Apr 18 22:35:02 2024 read: IOPS=37, BW=151KiB/s (155kB/s)(6028KiB/39883msec) clat (usec): min=1640, max=60511, avg=3585.21, stdev=3754.08 lat (usec): min=1641, max=60512, avg=3585.88, stdev=3754.10 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1860], 10.00th=[ 1909], 20.00th=[ 2008], | 30.00th=[ 2089], 40.00th=[ 2212], 50.00th=[ 2311], 60.00th=[ 2474], | 70.00th=[ 2737], 80.00th=[ 3490], 90.00th=[ 7439], 95.00th=[10290], | 99.00th=[20579], 99.50th=[25035], 99.90th=[36963], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 48, max= 280, per=25.20%, avg=151.23, stdev=55.80, samples=79 iops : min= 12, max= 70, avg=37.76, stdev=13.94, samples=79 write: IOPS=39, BW=157KiB/s (161kB/s)(6260KiB/39883msec) clat (msec): min=11, max=201, avg=21.96, stdev= 8.69 lat (msec): min=11, max=201, avg=21.96, stdev= 8.69 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 33], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 72], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 56, max= 208, per=24.97%, avg=156.80, stdev=43.43, samples=79 iops : min= 14, max= 52, avg=39.15, stdev=10.86, samples=79 lat (msec) : 2=9.60%, 4=30.47%, 10=6.22%, 20=24.35%, 50=29.07% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.09%, sys=4.50%, ctx=3315, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31724: Thu Apr 18 22:35:02 2024 read: IOPS=37, BW=151KiB/s (155kB/s)(6052KiB/39964msec) clat (usec): min=1588, max=46957, avg=3466.46, stdev=3603.05 lat (usec): min=1589, max=46958, avg=3467.14, stdev=3603.07 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1991], | 30.00th=[ 2089], 40.00th=[ 2180], 50.00th=[ 2278], 60.00th=[ 2442], | 70.00th=[ 2737], 80.00th=[ 3392], 90.00th=[ 6849], 95.00th=[ 9634], | 99.00th=[17433], 99.50th=[23200], 99.90th=[45876], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 32, max= 304, per=25.10%, avg=150.61, stdev=59.90, samples=79 iops : min= 8, max= 76, avg=37.59, stdev=15.00, samples=79 write: IOPS=39, BW=156KiB/s (160kB/s)(6236KiB/39964msec) clat (msec): min=11, max=199, avg=22.20, stdev= 8.69 lat (msec): min=11, max=199, avg=22.20, stdev= 8.69 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 33], 95.00th=[ 39], | 99.00th=[ 47], 99.50th=[ 54], 99.90th=[ 63], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 56, max= 208, per=24.80%, avg=155.77, stdev=42.32, samples=79 iops : min= 14, max= 52, avg=38.89, stdev=10.59, samples=79 lat (msec) : 2=10.51%, 4=30.73%, 10=5.66%, 20=23.40%, 50=29.36% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.12%, sys=4.36%, ctx=3346, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=600KiB/s (614kB/s), 148KiB/s-151KiB/s (152kB/s-155kB/s), io=23.4MiB (24.6MB), run=39803-40020msec WRITE: bw=628KiB/s (643kB/s), 156KiB/s-159KiB/s (160kB/s-162kB/s), io=24.6MiB (25.7MB), run=39803-40020msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31771: Thu Apr 18 22:35:07 2024 read: IOPS=87, BW=1404KiB/s (1438kB/s)(5920KiB/4217msec) clat (usec): min=51, max=29636, avg=8216.03, stdev=5241.57 lat (usec): min=52, max=29637, avg=8218.84, stdev=5241.62 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 64], 10.00th=[ 74], 20.00th=[ 100], | 30.00th=[ 7701], 40.00th=[ 8717], 50.00th=[ 9372], 60.00th=[10290], | 70.00th=[10814], 80.00th=[11731], 90.00th=[12911], 95.00th=[14353], | 99.00th=[23725], 99.50th=[28181], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 1120, max= 1600, per=23.86%, avg=1335.62, stdev=156.78, samples=8 iops : min= 70, max= 100, avg=83.38, stdev= 9.87, samples=8 write: IOPS=94, BW=1510KiB/s (1546kB/s)(6368KiB/4217msec) clat (usec): min=669, max=19918, avg=2940.87, stdev=2967.57 lat (usec): min=670, max=19919, avg=2941.71, stdev=2967.58 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 766], 20.00th=[ 799], | 30.00th=[ 840], 40.00th=[ 938], 50.00th=[ 2180], 60.00th=[ 2802], | 70.00th=[ 3392], 80.00th=[ 4948], 90.00th=[ 6456], 95.00th=[ 8848], | 99.00th=[15664], 99.50th=[17957], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1056, max= 1756, per=24.85%, avg=1463.50, stdev=231.14, samples=8 iops : min= 66, max= 109, avg=91.38, stdev=14.31, samples=8 lat (usec) : 100=9.64%, 250=1.04%, 500=0.39%, 750=3.91%, 1000=18.23% lat (msec) : 2=3.78%, 4=13.54%, 10=26.56%, 20=22.27%, 50=0.65% cpu : usr=0.12%, sys=22.22%, ctx=1960, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31772: Thu Apr 18 22:35:07 2024 read: IOPS=85, BW=1369KiB/s (1402kB/s)(5744KiB/4195msec) clat (usec): min=51, max=19198, avg=8331.57, stdev=4963.03 lat (usec): min=51, max=19199, avg=8332.15, stdev=4963.06 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 72], 10.00th=[ 75], 20.00th=[ 130], | 30.00th=[ 7635], 40.00th=[ 8717], 50.00th=[ 9634], 60.00th=[10552], | 70.00th=[11207], 80.00th=[11994], 90.00th=[13698], 95.00th=[14877], | 99.00th=[17171], 99.50th=[19268], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 992, max= 1792, per=24.29%, avg=1359.62, stdev=260.51, samples=8 iops : min= 62, max= 112, avg=84.88, stdev=16.28, samples=8 write: IOPS=97, BW=1560KiB/s (1597kB/s)(6544KiB/4195msec) clat (usec): min=647, max=54239, avg=2930.16, stdev=4047.37 lat (usec): min=648, max=54240, avg=2931.04, stdev=4047.45 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 799], | 30.00th=[ 840], 40.00th=[ 906], 50.00th=[ 1352], 60.00th=[ 2507], | 70.00th=[ 3195], 80.00th=[ 4113], 90.00th=[ 6390], 95.00th=[ 9241], | 99.00th=[16581], 99.50th=[21365], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 1152, max= 2304, per=25.94%, avg=1527.62, stdev=376.66, samples=8 iops : min= 72, max= 144, avg=95.38, stdev=23.56, samples=8 lat (usec) : 100=8.59%, 250=1.17%, 500=0.13%, 750=5.86%, 1000=18.62% lat (msec) : 2=4.95%, 4=13.54%, 10=23.05%, 20=23.70%, 50=0.26% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=21.89%, ctx=1909, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31773: Thu Apr 18 22:35:07 2024 read: IOPS=91, BW=1462KiB/s (1497kB/s)(6256KiB/4278msec) clat (usec): min=50, max=39151, avg=8221.09, stdev=5422.80 lat (usec): min=50, max=39152, avg=8221.63, stdev=5422.80 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 67], 10.00th=[ 74], 20.00th=[ 91], | 30.00th=[ 7570], 40.00th=[ 8717], 50.00th=[ 9634], 60.00th=[10421], | 70.00th=[11076], 80.00th=[11863], 90.00th=[13173], 95.00th=[15401], | 99.00th=[20841], 99.50th=[24773], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 1088, max= 1984, per=25.43%, avg=1423.62, stdev=317.69, samples=8 iops : min= 68, max= 124, avg=88.87, stdev=19.85, samples=8 write: IOPS=88, BW=1410KiB/s (1444kB/s)(6032KiB/4278msec) clat (usec): min=654, max=47528, avg=2804.73, stdev=3676.79 lat (usec): min=655, max=47528, avg=2805.73, stdev=3676.79 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 709], 10.00th=[ 734], 20.00th=[ 791], | 30.00th=[ 824], 40.00th=[ 881], 50.00th=[ 1287], 60.00th=[ 2507], | 70.00th=[ 3294], 80.00th=[ 4359], 90.00th=[ 5932], 95.00th=[ 7570], | 99.00th=[17695], 99.50th=[18744], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 960, max= 1952, per=23.02%, avg=1355.62, stdev=329.21, samples=8 iops : min= 60, max= 122, avg=84.62, stdev=20.64, samples=8 lat (usec) : 100=10.81%, 250=1.43%, 500=0.39%, 750=6.38%, 1000=16.54% lat (msec) : 2=4.17%, 4=10.68%, 10=25.39%, 20=23.57%, 50=0.65% cpu : usr=0.12%, sys=21.74%, ctx=1901, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31774: Thu Apr 18 22:35:07 2024 read: IOPS=88, BW=1416KiB/s (1450kB/s)(6032KiB/4259msec) clat (usec): min=48, max=32919, avg=8427.41, stdev=5304.13 lat (usec): min=48, max=32919, avg=8428.06, stdev=5304.23 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 72], 10.00th=[ 74], 20.00th=[ 88], | 30.00th=[ 7701], 40.00th=[ 8717], 50.00th=[ 9765], 60.00th=[10421], | 70.00th=[10945], 80.00th=[11994], 90.00th=[13566], 95.00th=[15401], | 99.00th=[21627], 99.50th=[27657], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 1088, max= 1568, per=24.43%, avg=1367.63, stdev=163.18, samples=8 iops : min= 68, max= 98, avg=85.38, stdev=10.27, samples=8 write: IOPS=91, BW=1469KiB/s (1504kB/s)(6256KiB/4259msec) clat (usec): min=667, max=49070, avg=2752.62, stdev=3730.14 lat (usec): min=668, max=49071, avg=2753.93, stdev=3729.91 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 832], 40.00th=[ 898], 50.00th=[ 1467], 60.00th=[ 2507], | 70.00th=[ 2933], 80.00th=[ 4228], 90.00th=[ 5800], 95.00th=[ 6980], | 99.00th=[18220], 99.50th=[30802], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 1088, max= 2016, per=24.44%, avg=1439.62, stdev=290.33, samples=8 iops : min= 68, max= 126, avg=89.88, stdev=18.16, samples=8 lat (usec) : 50=0.26%, 100=10.29%, 250=0.26%, 500=0.13%, 750=5.08% lat (usec) : 1000=17.84% lat (msec) : 2=3.65%, 4=14.19%, 10=24.74%, 20=22.79%, 50=0.78% cpu : usr=0.16%, sys=21.58%, ctx=1924, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5599KiB/s (5733kB/s), 1369KiB/s-1462KiB/s (1402kB/s-1497kB/s), io=23.4MiB (24.5MB), run=4195-4278msec WRITE: bw=5891KiB/s (6032kB/s), 1410KiB/s-1560KiB/s (1444kB/s-1597kB/s), io=24.6MiB (25.8MB), run=4195-4278msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31775: Thu Apr 18 22:35:13 2024 read: IOPS=37, BW=598KiB/s (612kB/s)(5920KiB/9907msec) clat (usec): min=1827, max=35433, avg=3824.42, stdev=3533.50 lat (usec): min=1827, max=35433, avg=3825.01, stdev=3533.51 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2343], 50.00th=[ 2507], 60.00th=[ 2769], | 70.00th=[ 3097], 80.00th=[ 4228], 90.00th=[ 7046], 95.00th=[10683], | 99.00th=[18482], 99.50th=[25297], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 288, max= 864, per=24.60%, avg=587.47, stdev=155.96, samples=19 iops : min= 18, max= 54, avg=36.53, stdev= 9.91, samples=19 write: IOPS=40, BW=643KiB/s (658kB/s)(6368KiB/9907msec) clat (usec): min=11588, max=48846, avg=21319.79, stdev=6398.22 lat (usec): min=11589, max=48847, avg=21320.80, stdev=6398.25 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13173], 10.00th=[13829], 20.00th=[15533], | 30.00th=[17171], 40.00th=[19006], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25560], 90.00th=[29754], 95.00th=[33162], | 99.00th=[42730], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 384, max= 800, per=25.27%, avg=634.68, stdev=152.89, samples=19 iops : min= 24, max= 50, avg=39.47, stdev= 9.68, samples=19 lat (msec) : 2=1.69%, 4=36.46%, 10=7.03%, 20=26.04%, 50=28.78% cpu : usr=0.08%, sys=5.08%, ctx=854, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31776: Thu Apr 18 22:35:13 2024 read: IOPS=35, BW=573KiB/s (586kB/s)(5744KiB/10029msec) clat (usec): min=1823, max=18528, avg=3613.04, stdev=2939.19 lat (usec): min=1824, max=18528, avg=3613.69, stdev=2939.17 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2376], 60.00th=[ 2606], | 70.00th=[ 2933], 80.00th=[ 4047], 90.00th=[ 7177], 95.00th=[10814], | 99.00th=[16712], 99.50th=[17695], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 192, max= 960, per=24.05%, avg=574.40, stdev=216.66, samples=20 iops : min= 12, max= 60, avg=35.90, stdev=13.54, samples=20 write: IOPS=40, BW=653KiB/s (668kB/s)(6544KiB/10029msec) clat (usec): min=11578, max=85499, avg=21328.84, stdev=7754.34 lat (usec): min=11579, max=85500, avg=21329.87, stdev=7754.36 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13042], 10.00th=[13566], 20.00th=[14877], | 30.00th=[16909], 40.00th=[18220], 50.00th=[20055], 60.00th=[21890], | 70.00th=[23200], 80.00th=[26084], 90.00th=[31065], 95.00th=[33817], | 99.00th=[46924], 99.50th=[48497], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 416, max= 896, per=25.92%, avg=651.20, stdev=161.93, samples=20 iops : min= 26, max= 56, avg=40.70, stdev=10.12, samples=20 lat (msec) : 2=1.04%, 4=36.33%, 10=6.51%, 20=29.30%, 50=26.56% lat (msec) : 100=0.26% cpu : usr=0.07%, sys=5.08%, ctx=861, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31777: Thu Apr 18 22:35:13 2024 read: IOPS=40, BW=651KiB/s (667kB/s)(6256KiB/9609msec) clat (usec): min=1957, max=25335, avg=3606.68, stdev=2940.95 lat (usec): min=1958, max=25335, avg=3607.26, stdev=2940.95 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2040], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2671], | 70.00th=[ 2966], 80.00th=[ 4146], 90.00th=[ 6521], 95.00th=[10552], | 99.00th=[15008], 99.50th=[22676], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 288, max= 1312, per=27.50%, avg=656.79, stdev=283.35, samples=19 iops : min= 18, max= 82, avg=41.00, stdev=17.77, samples=19 write: IOPS=39, BW=628KiB/s (643kB/s)(6032KiB/9609msec) clat (usec): min=11942, max=51687, avg=21727.08, stdev=6449.04 lat (usec): min=11942, max=51688, avg=21728.04, stdev=6449.10 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13435], 10.00th=[13960], 20.00th=[15664], | 30.00th=[17695], 40.00th=[20055], 50.00th=[21365], 60.00th=[22414], | 70.00th=[23462], 80.00th=[25822], 90.00th=[30540], 95.00th=[34341], | 99.00th=[43254], 99.50th=[43779], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 448, max= 800, per=24.81%, avg=623.11, stdev=136.81, samples=19 iops : min= 28, max= 50, avg=38.89, stdev= 8.61, samples=19 lat (msec) : 2=1.43%, 4=38.93%, 10=7.68%, 20=21.61%, 50=30.21% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=5.19%, ctx=844, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31778: Thu Apr 18 22:35:13 2024 read: IOPS=38, BW=610KiB/s (624kB/s)(6032KiB/9893msec) clat (usec): min=1950, max=21775, avg=3628.28, stdev=3003.34 lat (usec): min=1950, max=21775, avg=3628.86, stdev=3003.34 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 2933], 80.00th=[ 4015], 90.00th=[ 6718], 95.00th=[11076], | 99.00th=[16712], 99.50th=[18482], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 320, max= 896, per=25.17%, avg=601.05, stdev=187.84, samples=19 iops : min= 20, max= 56, avg=37.42, stdev=11.87, samples=19 write: IOPS=39, BW=632KiB/s (648kB/s)(6256KiB/9893msec) clat (usec): min=11904, max=65599, avg=21787.59, stdev=6895.52 lat (usec): min=11905, max=65600, avg=21788.60, stdev=6895.56 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13304], 10.00th=[13960], 20.00th=[15401], | 30.00th=[17433], 40.00th=[19792], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23725], 80.00th=[26346], 90.00th=[31589], 95.00th=[34866], | 99.00th=[41681], 99.50th=[45876], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 384, max= 832, per=25.00%, avg=627.89, stdev=154.38, samples=19 iops : min= 24, max= 52, avg=39.11, stdev= 9.69, samples=19 lat (msec) : 2=1.04%, 4=38.15%, 10=7.03%, 20=24.48%, 50=29.17% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=5.12%, ctx=861, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2388KiB/s (2446kB/s), 573KiB/s-651KiB/s (586kB/s-667kB/s), io=23.4MiB (24.5MB), run=9609-10029msec WRITE: bw=2513KiB/s (2573kB/s), 628KiB/s-653KiB/s (643kB/s-668kB/s), io=24.6MiB (25.8MB), run=9609-10029msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31792: Thu Apr 18 22:35:17 2024 read: IOPS=1, BW=1856KiB/s (1900kB/s)(7168KiB/3863msec) clat (usec): min=344, max=737078, avg=430538.54, stdev=288359.81 lat (usec): min=344, max=737080, avg=430539.45, stdev=288360.04 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 44827], | 30.00th=[517997], 40.00th=[517997], 50.00th=[534774], 60.00th=[557843], | 70.00th=[557843], 80.00th=[624952], 90.00th=[734004], 95.00th=[734004], | 99.00th=[734004], 99.50th=[734004], 99.90th=[734004], 99.95th=[734004], | 99.99th=[734004] bw ( KiB/s): min= 2048, max= 2048, per=38.63%, avg=2048.00, stdev= 0.00, samples=6 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=6 write: IOPS=1, BW=1325KiB/s (1357kB/s)(5120KiB/3863msec) clat (msec): min=23, max=642, avg=169.59, stdev=267.88 lat (msec): min=23, max=642, avg=169.64, stdev=267.87 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 6144, per=45.99%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=33.33%, 250=8.33%, 750=50.00% cpu : usr=0.00%, sys=15.74%, ctx=1430, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31793: Thu Apr 18 22:35:17 2024 read: IOPS=0, BW=635KiB/s (650kB/s)(2048KiB/3226msec) clat (msec): min=50, max=494, avg=272.49, stdev=314.32 lat (msec): min=50, max=494, avg=272.49, stdev=314.31 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 2048, max= 2048, per=38.63%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3174KiB/s (3250kB/s)(10.0MiB/3226msec) clat (msec): min=19, max=1412, avg=267.98, stdev=515.39 lat (msec): min=19, max=1412, avg=268.02, stdev=515.38 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 22], 40.00th=[ 23], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 1053], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 4087, max= 6144, per=64.34%, avg=4775.67, stdev=1185.02, samples=3 iops : min= 3, max= 6, avg= 4.33, stdev= 1.53, samples=3 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 500=8.33% cpu : usr=0.00%, sys=8.93%, ctx=350, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31794: Thu Apr 18 22:35:17 2024 read: IOPS=1, BW=1601KiB/s (1639kB/s)(6144KiB/3838msec) clat (usec): min=317, max=657059, avg=298209.35, stdev=311838.39 lat (usec): min=317, max=657060, avg=298210.07, stdev=311838.41 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 318], 20.00th=[ 545], | 30.00th=[ 545], 40.00th=[ 49546], 50.00th=[ 49546], 60.00th=[526386], | 70.00th=[557843], 80.00th=[557843], 90.00th=[658506], 95.00th=[658506], | 99.00th=[658506], 99.50th=[658506], 99.90th=[658506], 99.95th=[658506], | 99.99th=[658506] bw ( KiB/s): min= 2048, max= 4096, per=64.39%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1601KiB/s (1639kB/s)(6144KiB/3838msec) clat (msec): min=20, max=1389, avg=341.30, stdev=541.95 lat (msec): min=20, max=1389, avg=341.33, stdev=541.94 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 79], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 2043, max= 4096, per=33.10%, avg=2456.60, stdev=916.46, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=25.00%, 100=16.67%, 500=8.33%, 750=25.00% cpu : usr=0.00%, sys=9.64%, ctx=833, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31795: Thu Apr 18 22:35:17 2024 read: IOPS=1, BW=1334KiB/s (1366kB/s)(5120KiB/3839msec) clat (msec): min=35, max=623, avg=425.53, stdev=232.34 lat (msec): min=35, max=623, avg=425.53, stdev=232.34 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 2048, max= 4096, per=51.51%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1867KiB/s (1912kB/s)(7168KiB/3839msec) clat (msec): min=17, max=1382, avg=244.18, stdev=503.26 lat (msec): min=17, max=1382, avg=244.22, stdev=503.26 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 49], 60.00th=[ 80], | 70.00th=[ 80], 80.00th=[ 116], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 2043, max= 6144, per=48.27%, avg=3582.75, stdev=1962.12, samples=4 iops : min= 1, max= 6, avg= 3.25, stdev= 2.22, samples=4 lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=8.33%, 500=16.67% lat (msec) : 750=16.67% cpu : usr=0.00%, sys=10.34%, ctx=1124, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5302KiB/s (5429kB/s), 635KiB/s-1856KiB/s (650kB/s-1900kB/s), io=20.0MiB (20.0MB), run=3226-3863msec WRITE: bw=7422KiB/s (7600kB/s), 1325KiB/s-3174KiB/s (1357kB/s-3250kB/s), io=28.0MiB (29.4MB), run=3226-3863msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31796: Thu Apr 18 22:35:17 2024 read: IOPS=1, BW=1849KiB/s (1893kB/s)(7168KiB/3877msec) clat (msec): min=26, max=729, avg=421.02, stdev=274.34 lat (msec): min=26, max=729, avg=421.02, stdev=274.34 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 51], | 30.00th=[ 493], 40.00th=[ 493], 50.00th=[ 498], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 625], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 2015, max= 2048, per=38.70%, avg=2041.67, stdev=13.22, samples=6 iops : min= 1, max= 2, avg= 1.67, stdev= 0.52, samples=6 write: IOPS=1, BW=1321KiB/s (1352kB/s)(5120KiB/3877msec) clat (msec): min=36, max=664, avg=185.72, stdev=268.65 lat (msec): min=37, max=664, avg=185.76, stdev=268.65 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 65], 60.00th=[ 65], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 6047, per=45.78%, avg=3381.00, stdev=2308.82, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=8.33%, 500=16.67%, 750=33.33% cpu : usr=0.03%, sys=3.38%, ctx=45, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31797: Thu Apr 18 22:35:17 2024 read: IOPS=0, BW=622KiB/s (637kB/s)(2048KiB/3294msec) clat (msec): min=106, max=285, avg=196.02, stdev=126.53 lat (msec): min=106, max=285, avg=196.02, stdev=126.53 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 2048, max= 2048, per=38.82%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3109KiB/s (3183kB/s)(10.0MiB/3294msec) clat (msec): min=27, max=1393, avg=289.89, stdev=496.86 lat (msec): min=27, max=1393, avg=289.93, stdev=496.85 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 36], 40.00th=[ 39], 50.00th=[ 59], 60.00th=[ 77], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 1045], 95.00th=[ 1401], | 99.00th=[ 1401], 99.50th=[ 1401], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 4096, max=10240, per=97.06%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 50=33.33%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=3.46%, ctx=98, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31798: Thu Apr 18 22:35:17 2024 read: IOPS=1, BW=1583KiB/s (1621kB/s)(6144KiB/3882msec) clat (msec): min=17, max=628, avg=286.26, stdev=281.15 lat (msec): min=17, max=628, avg=286.26, stdev=281.15 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 485], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 2048, max= 4096, per=64.65%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=1, BW=1583KiB/s (1621kB/s)(6144KiB/3882msec) clat (msec): min=37, max=1382, avg=360.19, stdev=517.73 lat (msec): min=37, max=1382, avg=360.22, stdev=517.72 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 167], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 2048, max= 4096, per=41.57%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (msec) : 20=8.33%, 50=33.33%, 250=16.67%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=2.78%, ctx=56, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31799: Thu Apr 18 22:35:17 2024 read: IOPS=1, BW=1333KiB/s (1365kB/s)(5120KiB/3842msec) clat (msec): min=17, max=1048, avg=391.49, stdev=414.09 lat (msec): min=17, max=1048, avg=391.49, stdev=414.09 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2048, max= 4096, per=51.77%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1866KiB/s (1910kB/s)(7168KiB/3842msec) clat (msec): min=35, max=1363, avg=268.23, stdev=485.05 lat (msec): min=35, max=1363, avg=268.27, stdev=485.04 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 91], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 146], 90.00th=[ 1368], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 2048, max= 6144, per=64.71%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=3.44%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5276KiB/s (5402kB/s), 622KiB/s-1849KiB/s (637kB/s-1893kB/s), io=20.0MiB (20.0MB), run=3294-3882msec WRITE: bw=7386KiB/s (7563kB/s), 1321KiB/s-3109KiB/s (1352kB/s-3183kB/s), io=28.0MiB (29.4MB), run=3294-3882msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31808: Thu Apr 18 22:35:20 2024 read: IOPS=3, BW=15.0MiB/s (15.7MB/s)(12.0MiB/800msec) clat (msec): min=26, max=712, avg=263.98, stdev=388.90 lat (msec): min=26, max=712, avg=263.99, stdev=388.90 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 8192, max= 8192, per=82.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=85.23%, ctx=89, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31810: Thu Apr 18 22:35:20 2024 read: IOPS=0, BW=2496KiB/s (2556kB/s)(4096KiB/1641msec) clat (nsec): min=70365k, max=70365k, avg=70365358.00, stdev= 0.00 lat (nsec): min=70368k, max=70368k, avg=70367514.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[70779], 5.00th=[70779], 10.00th=[70779], 20.00th=[70779], | 30.00th=[70779], 40.00th=[70779], 50.00th=[70779], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 8192, max= 8192, per=82.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4992KiB/s (5112kB/s)(8192KiB/1641msec) clat (msec): min=508, max=1050, avg=779.13, stdev=383.06 lat (msec): min=508, max=1050, avg=779.29, stdev=383.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 1053], | 70.00th=[ 1053], 80.00th=[ 1053], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 8192, max= 8192, per=41.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=13.84%, ctx=607, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31812: Thu Apr 18 22:35:20 2024 write: IOPS=2, BW=9029KiB/s (9245kB/s)(12.0MiB/1361msec) clat (msec): min=158, max=992, avg=450.91, stdev=469.72 lat (msec): min=158, max=992, avg=451.18, stdev=469.71 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 995], 80.00th=[ 995], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 8192, max= 8192, per=41.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=21.18%, ctx=68, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31814: Thu Apr 18 22:35:20 2024 write: IOPS=1, BW=7758KiB/s (7944kB/s)(12.0MiB/1584msec) clat (msec): min=91, max=821, avg=522.77, stdev=382.34 lat (msec): min=92, max=821, avg=523.02, stdev=382.34 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8192, max= 8192, per=41.03%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=16.11%, ctx=66, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9984KiB/s (10.2MB/s), 2496KiB/s-15.0MiB/s (2556kB/s-15.7MB/s), io=16.0MiB (16.8MB), run=800-1641msec WRITE: bw=19.5MiB/s (20.4MB/s), 4992KiB/s-9029KiB/s (5112kB/s-9245kB/s), io=32.0MiB (33.6MB), run=1361-1641msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31809: Thu Apr 18 22:35:20 2024 read: IOPS=2, BW=8815KiB/s (9026kB/s)(12.0MiB/1394msec) clat (msec): min=135, max=977, avg=464.19, stdev=450.64 lat (msec): min=135, max=977, avg=464.20, stdev=450.64 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 8192, max= 8192, per=87.16%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.07%, sys=6.32%, ctx=27, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31811: Thu Apr 18 22:35:20 2024 read: IOPS=0, BW=2350KiB/s (2406kB/s)(4096KiB/1743msec) clat (nsec): min=243251k, max=243251k, avg=243251461.00, stdev= 0.00 lat (nsec): min=243253k, max=243253k, avg=243253235.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 243], | 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 243], 80.00th=[ 243], 90.00th=[ 243], 95.00th=[ 243], | 99.00th=[ 243], 99.50th=[ 243], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 8192, max= 8192, per=87.16%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4700KiB/s (4813kB/s)(8192KiB/1743msec) clat (msec): min=525, max=973, avg=749.34, stdev=316.74 lat (msec): min=525, max=973, avg=749.57, stdev=316.60 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 978], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7968, max= 7968, per=44.40%, avg=7968.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=6.14%, ctx=64, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31813: Thu Apr 18 22:35:20 2024 write: IOPS=2, BW=8463KiB/s (8666kB/s)(12.0MiB/1452msec) clat (msec): min=118, max=1004, avg=483.44, stdev=463.44 lat (msec): min=118, max=1005, avg=483.64, stdev=463.44 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 8192, max= 8192, per=45.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.07%, sys=6.27%, ctx=20, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31815: Thu Apr 18 22:35:20 2024 write: IOPS=1, BW=6729KiB/s (6891kB/s)(12.0MiB/1826msec) clat (msec): min=106, max=977, avg=606.89, stdev=449.44 lat (msec): min=107, max=977, avg=607.18, stdev=449.42 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 735], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 8192, max= 8192, per=45.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=6.85%, ctx=60, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9400KiB/s (9625kB/s), 2350KiB/s-8815KiB/s (2406kB/s-9026kB/s), io=16.0MiB (16.8MB), run=1394-1743msec WRITE: bw=17.5MiB/s (18.4MB/s), 4700KiB/s-8463KiB/s (4813kB/s-8666kB/s), io=32.0MiB (33.6MB), run=1452-1826msec