== sanity test 398c: run fio to test AIO ================= 13:15:21 (1713287721) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.145925 s, 287 MB/s osc.lustre-OST0000-osc-ffff88012b766800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=31245: Tue Apr 16 13:15:58 2024 write: IOPS=73, BW=295KiB/s (302kB/s)(10.0MiB/34701msec) slat (usec): min=13, max=222, avg=48.24, stdev=26.89 clat (msec): min=43, max=450, avg=216.63, stdev=38.60 lat (msec): min=43, max=450, avg=216.68, stdev=38.60 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 159], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 220], 60.00th=[ 230], | 70.00th=[ 239], 80.00th=[ 249], 90.00th=[ 264], 95.00th=[ 275], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 439], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 224, max= 416, per=25.04%, avg=295.00, stdev=39.59, samples=69 iops : min= 56, max= 104, avg=73.70, stdev= 9.85, samples=69 lat (msec) : 50=0.16%, 100=0.20%, 250=81.72%, 500=17.93% cpu : usr=0.10%, sys=0.45%, ctx=2129, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31246: Tue Apr 16 13:15:58 2024 write: IOPS=73, BW=295KiB/s (302kB/s)(10.0MiB/34745msec) slat (usec): min=11, max=276, avg=49.80, stdev=29.24 clat (msec): min=115, max=416, avg=216.82, stdev=36.94 lat (msec): min=115, max=416, avg=216.87, stdev=36.94 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 157], 10.00th=[ 169], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 207], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 236], 80.00th=[ 247], 90.00th=[ 262], 95.00th=[ 271], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 232, max= 384, per=24.97%, avg=294.10, stdev=38.26, samples=69 iops : min= 58, max= 96, avg=73.49, stdev= 9.54, samples=69 lat (msec) : 250=83.01%, 500=16.99% cpu : usr=0.13%, sys=0.39%, ctx=2377, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31247: Tue Apr 16 13:15:58 2024 write: IOPS=73, BW=295KiB/s (302kB/s)(10.0MiB/34745msec) slat (usec): min=12, max=231, avg=49.83, stdev=28.88 clat (msec): min=115, max=325, avg=216.85, stdev=36.13 lat (msec): min=115, max=326, avg=216.90, stdev=36.13 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 159], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 249], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 326] bw ( KiB/s): min= 216, max= 384, per=24.99%, avg=294.36, stdev=40.92, samples=69 iops : min= 54, max= 96, avg=73.58, stdev=10.23, samples=69 lat (msec) : 250=82.46%, 500=17.54% cpu : usr=0.07%, sys=0.45%, ctx=2355, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31248: Tue Apr 16 13:15:58 2024 write: IOPS=73, BW=295KiB/s (302kB/s)(10.0MiB/34764msec) slat (usec): min=12, max=534, avg=49.91, stdev=30.55 clat (msec): min=107, max=318, avg=216.98, stdev=35.63 lat (msec): min=107, max=318, avg=217.03, stdev=35.63 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 159], 10.00th=[ 171], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 249], 90.00th=[ 262], 95.00th=[ 271], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 232, max= 384, per=24.94%, avg=293.78, stdev=37.66, samples=69 iops : min= 58, max= 96, avg=73.43, stdev= 9.41, samples=69 lat (msec) : 250=81.76%, 500=18.24% cpu : usr=0.09%, sys=0.43%, ctx=2433, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1178KiB/s (1207kB/s), 295KiB/s-295KiB/s (302kB/s-302kB/s), io=40.0MiB (41.9MB), run=34701-34764msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31286: Tue Apr 16 13:16:21 2024 read: IOPS=56, BW=226KiB/s (231kB/s)(5048KiB/22339msec) slat (usec): min=31, max=3481, avg=1351.52, stdev=295.49 clat (usec): min=1597, max=218858, avg=92523.79, stdev=19571.13 lat (msec): min=2, max=220, avg=93.88, stdev=19.64 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 97], | 70.00th=[ 102], 80.00th=[ 108], 90.00th=[ 115], 95.00th=[ 123], | 99.00th=[ 136], 99.50th=[ 146], 99.90th=[ 192], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 128, max= 352, per=25.03%, avg=224.77, stdev=48.71, samples=44 iops : min= 32, max= 88, avg=56.09, stdev=12.16, samples=44 write: IOPS=58, BW=232KiB/s (238kB/s)(5192KiB/22339msec) slat (usec): min=13, max=311, avg=46.79, stdev=24.10 clat (msec): min=53, max=329, avg=183.95, stdev=42.41 lat (msec): min=53, max=329, avg=184.00, stdev=42.41 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 239], 95.00th=[ 253], | 99.00th=[ 300], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 191, max= 288, per=24.74%, avg=231.32, stdev=28.66, samples=44 iops : min= 47, max= 72, avg=57.73, stdev= 7.16, samples=44 lat (msec) : 2=0.08%, 4=0.04%, 10=0.12%, 20=0.35%, 50=0.08% lat (msec) : 100=33.24%, 250=63.28%, 500=2.81% cpu : usr=0.14%, sys=0.97%, ctx=3562, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31287: Tue Apr 16 13:16:21 2024 read: IOPS=55, BW=223KiB/s (228kB/s)(4976KiB/22311msec) slat (usec): min=27, max=3432, avg=1366.59, stdev=290.46 clat (msec): min=6, max=192, avg=92.37, stdev=19.26 lat (msec): min=6, max=194, avg=93.74, stdev=19.31 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 108], 90.00th=[ 114], 95.00th=[ 122], | 99.00th=[ 142], 99.50th=[ 155], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 128, max= 360, per=24.70%, avg=221.80, stdev=51.80, samples=44 iops : min= 32, max= 90, avg=55.43, stdev=12.95, samples=44 write: IOPS=58, BW=236KiB/s (242kB/s)(5264KiB/22311msec) slat (usec): min=13, max=207, avg=46.80, stdev=23.89 clat (msec): min=53, max=340, avg=182.52, stdev=43.00 lat (msec): min=53, max=340, avg=182.56, stdev=43.00 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 116], 10.00th=[ 128], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 251], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 168, max= 304, per=25.16%, avg=235.25, stdev=31.50, samples=44 iops : min= 42, max= 76, avg=58.80, stdev= 7.84, samples=44 lat (msec) : 10=0.20%, 20=0.20%, 50=0.31%, 100=33.32%, 250=63.20% lat (msec) : 500=2.77% cpu : usr=0.13%, sys=1.00%, ctx=3225, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31288: Tue Apr 16 13:16:21 2024 read: IOPS=55, BW=223KiB/s (229kB/s)(4984KiB/22310msec) slat (usec): min=30, max=3378, avg=1363.80, stdev=297.47 clat (usec): min=1624, max=244826, avg=92326.45, stdev=20350.14 lat (msec): min=2, max=246, avg=93.69, stdev=20.41 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 97], | 70.00th=[ 102], 80.00th=[ 108], 90.00th=[ 115], 95.00th=[ 122], | 99.00th=[ 136], 99.50th=[ 140], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 128, max= 336, per=24.98%, avg=224.34, stdev=49.51, samples=44 iops : min= 32, max= 84, avg=56.07, stdev=12.39, samples=44 write: IOPS=58, BW=236KiB/s (241kB/s)(5256KiB/22310msec) slat (usec): min=13, max=273, avg=46.77, stdev=23.50 clat (msec): min=53, max=328, avg=182.61, stdev=42.46 lat (msec): min=53, max=328, avg=182.66, stdev=42.46 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 116], 10.00th=[ 130], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 305], 99.50th=[ 309], 99.90th=[ 321], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 168, max= 312, per=25.12%, avg=234.89, stdev=36.07, samples=44 iops : min= 42, max= 78, avg=58.70, stdev= 8.98, samples=44 lat (msec) : 2=0.04%, 4=0.04%, 10=0.12%, 20=0.51%, 50=0.20% lat (msec) : 100=32.93%, 250=63.12%, 500=3.05% cpu : usr=0.11%, sys=1.01%, ctx=3306, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31289: Tue Apr 16 13:16:21 2024 read: IOPS=57, BW=229KiB/s (234kB/s)(5060KiB/22102msec) slat (usec): min=27, max=3265, avg=1356.19, stdev=297.27 clat (msec): min=2, max=186, avg=91.67, stdev=19.10 lat (msec): min=4, max=188, avg=93.03, stdev=19.16 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 65], 10.00th=[ 72], 20.00th=[ 78], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 107], 90.00th=[ 115], 95.00th=[ 122], | 99.00th=[ 136], 99.50th=[ 138], 99.90th=[ 167], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 128, max= 344, per=25.53%, avg=229.27, stdev=47.53, samples=44 iops : min= 32, max= 86, avg=57.32, stdev=11.88, samples=44 write: IOPS=58, BW=234KiB/s (240kB/s)(5180KiB/22102msec) slat (usec): min=12, max=984, avg=48.28, stdev=36.69 clat (msec): min=44, max=356, avg=182.02, stdev=43.04 lat (msec): min=45, max=356, avg=182.07, stdev=43.03 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 163], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 313], 99.50th=[ 330], 99.90th=[ 347], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 160, max= 304, per=24.97%, avg=233.45, stdev=30.29, samples=44 iops : min= 40, max= 76, avg=58.36, stdev= 7.57, samples=44 lat (msec) : 4=0.04%, 10=0.16%, 20=0.23%, 50=0.31%, 100=33.67% lat (msec) : 250=62.46%, 500=3.12% cpu : usr=0.14%, sys=0.98%, ctx=3663, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=898KiB/s (920kB/s), 223KiB/s-229KiB/s (228kB/s-234kB/s), io=19.6MiB (20.5MB), run=22102-22339msec WRITE: bw=935KiB/s (958kB/s), 232KiB/s-236KiB/s (238kB/s-242kB/s), io=20.4MiB (21.4MB), run=22102-22339msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=31310: Tue Apr 16 13:16:22 2024 read: IOPS=7, BW=301MiB/s (315MB/s)(40.0MiB/133msec) slat (nsec): min=6653.8k, max=6653.8k, avg=6653824.00, stdev= 0.00 clat (nsec): min=125259k, max=125259k, avg=125259447.00, stdev= 0.00 lat (nsec): min=131923k, max=131923k, avg=131922674.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 126], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.55%, ctx=17, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=301MiB/s (315MB/s), 301MiB/s-301MiB/s (315MB/s-315MB/s), io=40.0MiB (41.9MB), run=133-133msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout