== sanity test 398c: run fio to test AIO ================= 15:47:10 (1713296830) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.104784 s, 400 MB/s osc.lustre-OST0000-osc-ffff8800a8439000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=31121: Tue Apr 16 15:47:42 2024 write: IOPS=83, BW=335KiB/s (343kB/s)(10.0MiB/30597msec) slat (usec): min=12, max=480, avg=75.54, stdev=44.59 clat (msec): min=18, max=367, avg=190.92, stdev=33.58 lat (msec): min=18, max=367, avg=191.00, stdev=33.59 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 218], 90.00th=[ 232], 95.00th=[ 243], | 99.00th=[ 268], 99.50th=[ 271], 99.90th=[ 355], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 264, max= 480, per=24.98%, avg=333.77, stdev=43.94, samples=61 iops : min= 66, max= 120, avg=83.41, stdev=10.97, samples=61 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=96.72%, 500=2.97% cpu : usr=0.15%, sys=0.75%, ctx=2437, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31122: Tue Apr 16 15:47:42 2024 write: IOPS=83, BW=335KiB/s (343kB/s)(10.0MiB/30597msec) slat (usec): min=12, max=469, avg=73.33, stdev=42.36 clat (msec): min=18, max=386, avg=190.92, stdev=33.73 lat (msec): min=18, max=386, avg=191.00, stdev=33.73 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 132], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 218], 90.00th=[ 232], 95.00th=[ 243], | 99.00th=[ 264], 99.50th=[ 275], 99.90th=[ 342], 99.95th=[ 376], | 99.99th=[ 388] bw ( KiB/s): min= 272, max= 480, per=24.97%, avg=333.64, stdev=42.92, samples=61 iops : min= 68, max= 120, avg=83.31, stdev=10.72, samples=61 lat (msec) : 20=0.04%, 100=0.23%, 250=96.80%, 500=2.93% cpu : usr=0.12%, sys=0.72%, ctx=2561, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31123: Tue Apr 16 15:47:42 2024 write: IOPS=83, BW=334KiB/s (342kB/s)(10.0MiB/30619msec) slat (usec): min=12, max=414, avg=73.26, stdev=42.08 clat (msec): min=50, max=314, avg=191.10, stdev=32.71 lat (msec): min=50, max=314, avg=191.17, stdev=32.71 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 218], 90.00th=[ 232], 95.00th=[ 243], | 99.00th=[ 262], 99.50th=[ 268], 99.90th=[ 305], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 264, max= 480, per=24.95%, avg=333.38, stdev=44.69, samples=61 iops : min= 66, max= 120, avg=83.25, stdev=11.17, samples=61 lat (msec) : 100=0.27%, 250=97.07%, 500=2.66% cpu : usr=0.14%, sys=0.72%, ctx=2525, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31124: Tue Apr 16 15:47:42 2024 write: IOPS=83, BW=334KiB/s (342kB/s)(10.0MiB/30642msec) slat (usec): min=12, max=587, avg=69.28, stdev=43.99 clat (msec): min=56, max=275, avg=191.26, stdev=32.20 lat (msec): min=56, max=275, avg=191.33, stdev=32.20 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 218], 90.00th=[ 232], 95.00th=[ 243], | 99.00th=[ 264], 99.50th=[ 268], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 256, max= 512, per=24.93%, avg=333.11, stdev=43.51, samples=61 iops : min= 64, max= 128, avg=83.18, stdev=10.87, samples=61 lat (msec) : 100=0.20%, 250=96.76%, 500=3.05% cpu : usr=0.12%, sys=0.68%, ctx=2340, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1337KiB/s (1369kB/s), 334KiB/s-335KiB/s (342kB/s-343kB/s), io=40.0MiB (41.9MB), run=30597-30642msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31158: Tue Apr 16 15:48:02 2024 read: IOPS=63, BW=254KiB/s (261kB/s)(5048KiB/19840msec) slat (usec): min=781, max=3705, avg=1421.88, stdev=374.27 clat (msec): min=2, max=160, avg=81.86, stdev=14.47 lat (msec): min=4, max=161, avg=83.29, stdev=14.50 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 62], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 89], 80.00th=[ 93], 90.00th=[ 101], 95.00th=[ 105], | 99.00th=[ 115], 99.50th=[ 124], 99.90th=[ 148], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 176, max= 336, per=25.05%, avg=253.26, stdev=45.17, samples=39 iops : min= 44, max= 84, avg=63.26, stdev=11.36, samples=39 write: IOPS=65, BW=262KiB/s (268kB/s)(5192KiB/19840msec) slat (usec): min=13, max=612, avg=65.28, stdev=41.25 clat (msec): min=65, max=295, avg=163.26, stdev=37.91 lat (msec): min=65, max=296, avg=163.33, stdev=37.91 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 211], 95.00th=[ 228], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 288], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 192, max= 344, per=24.78%, avg=260.64, stdev=37.25, samples=39 iops : min= 48, max= 86, avg=65.10, stdev= 9.32, samples=39 lat (msec) : 4=0.08%, 10=0.04%, 20=0.12%, 50=0.62%, 100=44.88% lat (msec) : 250=53.52%, 500=0.74% cpu : usr=0.22%, sys=1.50%, ctx=2988, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31159: Tue Apr 16 15:48:02 2024 read: IOPS=62, BW=251KiB/s (257kB/s)(4976KiB/19849msec) slat (usec): min=11, max=3362, avg=1407.55, stdev=387.70 clat (usec): min=1782, max=124587, avg=80648.99, stdev=15076.99 lat (usec): min=1877, max=126285, avg=82058.80, stdev=15111.77 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 89], 80.00th=[ 92], 90.00th=[ 99], 95.00th=[ 104], | 99.00th=[ 114], 99.50th=[ 116], 99.90th=[ 124], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 152, max= 384, per=24.70%, avg=249.74, stdev=56.11, samples=39 iops : min= 38, max= 96, avg=62.36, stdev=14.10, samples=39 write: IOPS=66, BW=265KiB/s (272kB/s)(5264KiB/19849msec) slat (usec): min=10, max=304, avg=64.82, stdev=36.36 clat (msec): min=18, max=305, avg=163.57, stdev=39.88 lat (msec): min=18, max=306, avg=163.64, stdev=39.89 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 230], | 99.00th=[ 262], 99.50th=[ 288], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 216, max= 344, per=25.01%, avg=263.08, stdev=34.42, samples=39 iops : min= 54, max= 86, avg=65.69, stdev= 8.64, samples=39 lat (msec) : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.35%, 50=0.62% lat (msec) : 100=45.27%, 250=52.38%, 500=1.21% cpu : usr=0.20%, sys=1.53%, ctx=2944, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31160: Tue Apr 16 15:48:02 2024 read: IOPS=62, BW=251KiB/s (257kB/s)(4984KiB/19827msec) slat (usec): min=642, max=3475, avg=1411.01, stdev=359.36 clat (msec): min=2, max=170, avg=80.93, stdev=15.94 lat (msec): min=4, max=171, avg=82.34, stdev=15.97 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 81], 60.00th=[ 86], | 70.00th=[ 89], 80.00th=[ 93], 90.00th=[ 100], 95.00th=[ 106], | 99.00th=[ 122], 99.50th=[ 125], 99.90th=[ 148], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 168, max= 408, per=24.93%, avg=252.00, stdev=44.74, samples=39 iops : min= 42, max= 102, avg=62.92, stdev=11.26, samples=39 write: IOPS=66, BW=265KiB/s (271kB/s)(5256KiB/19827msec) slat (usec): min=12, max=530, avg=65.84, stdev=40.50 clat (msec): min=79, max=304, avg=163.04, stdev=37.63 lat (msec): min=79, max=304, avg=163.11, stdev=37.64 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 213], 95.00th=[ 228], | 99.00th=[ 255], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 184, max= 336, per=25.01%, avg=263.08, stdev=40.84, samples=39 iops : min= 46, max= 84, avg=65.69, stdev=10.26, samples=39 lat (msec) : 4=0.04%, 10=0.08%, 20=0.35%, 50=0.55%, 100=44.26% lat (msec) : 250=53.95%, 500=0.78% cpu : usr=0.22%, sys=1.50%, ctx=3025, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31161: Tue Apr 16 15:48:02 2024 read: IOPS=64, BW=257KiB/s (263kB/s)(5060KiB/19704msec) slat (usec): min=806, max=4174, avg=1409.33, stdev=362.76 clat (msec): min=2, max=200, avg=81.23, stdev=15.78 lat (msec): min=3, max=201, avg=82.64, stdev=15.80 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 89], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 106], | 99.00th=[ 122], 99.50th=[ 125], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 152, max= 344, per=25.52%, avg=257.97, stdev=44.42, samples=39 iops : min= 38, max= 86, avg=64.44, stdev=11.14, samples=39 write: IOPS=65, BW=263KiB/s (269kB/s)(5180KiB/19704msec) slat (usec): min=12, max=2029, avg=66.63, stdev=67.96 clat (msec): min=62, max=300, avg=162.39, stdev=37.72 lat (msec): min=62, max=300, avg=162.46, stdev=37.72 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 213], 95.00th=[ 226], | 99.00th=[ 266], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 200, max= 352, per=24.89%, avg=261.87, stdev=38.15, samples=39 iops : min= 50, max= 88, avg=65.41, stdev= 9.59, samples=39 lat (msec) : 4=0.04%, 20=0.23%, 50=0.55%, 100=45.55%, 250=52.77% lat (msec) : 500=0.86% cpu : usr=0.23%, sys=1.51%, ctx=3077, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1011KiB/s (1035kB/s), 251KiB/s-257KiB/s (257kB/s-263kB/s), io=19.6MiB (20.5MB), run=19704-19849msec WRITE: bw=1053KiB/s (1078kB/s), 262KiB/s-265KiB/s (268kB/s-272kB/s), io=20.4MiB (21.4MB), run=19704-19849msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=31181: Tue Apr 16 15:48:04 2024 read: IOPS=9, BW=377MiB/s (396MB/s)(40.0MiB/106msec) slat (nsec): min=6741.8k, max=6741.8k, avg=6741839.00, stdev= 0.00 clat (nsec): min=93800k, max=93800k, avg=93800039.00, stdev= 0.00 lat (nsec): min=100566k, max=100566k, avg=100565925.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[93848], 5.00th=[93848], 10.00th=[93848], 20.00th=[93848], | 30.00th=[93848], 40.00th=[93848], 50.00th=[93848], 60.00th=[93848], | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=6.67%, ctx=4, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=40.0MiB (41.9MB), run=106-106msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout