== sanity test 398c: run fio to test AIO ================= 17:33:25 (1713303205)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0872678 s, 481 MB/s
osc.lustre-OST0000-osc-ffff88012ab49000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=13000: Tue Apr 16 17:33:30 2024
  write: IOPS=661, BW=2646KiB/s (2709kB/s)(10.0MiB/3870msec)
    slat (usec): min=11, max=1128, avg=46.71, stdev=34.17
    clat (usec): min=10460, max=49399, avg=24035.46, stdev=5849.20
     lat (usec): min=10544, max=49442, avg=24083.51, stdev=5856.17
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[17171], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22676], 60.00th=[23725],
     | 70.00th=[24511], 80.00th=[26870], 90.00th=[31065], 95.00th=[36963],
     | 99.00th=[44827], 99.50th=[45351], 99.90th=[45876], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min= 2088, max= 3024, per=24.83%, avg=2627.71, stdev=305.63, samples=7
   iops        : min=  522, max=  756, avg=656.86, stdev=76.38, samples=7
  lat (msec)   : 20=18.59%, 50=81.41%
  cpu          : usr=0.59%, sys=3.85%, ctx=1335, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13001: Tue Apr 16 17:33:30 2024
  write: IOPS=662, BW=2648KiB/s (2712kB/s)(10.0MiB/3867msec)
    slat (usec): min=12, max=845, avg=46.50, stdev=30.78
    clat (usec): min=10485, max=60777, avg=24012.65, stdev=5846.21
     lat (usec): min=10541, max=60844, avg=24060.35, stdev=5852.37
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[17171], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22938], 60.00th=[23462],
     | 70.00th=[24773], 80.00th=[26870], 90.00th=[31327], 95.00th=[36963],
     | 99.00th=[43779], 99.50th=[45351], 99.90th=[45876], 99.95th=[45876],
     | 99.99th=[60556]
   bw (  KiB/s): min= 2104, max= 3056, per=24.86%, avg=2631.14, stdev=308.85, samples=7
   iops        : min=  526, max=  764, avg=657.71, stdev=77.19, samples=7
  lat (msec)   : 20=18.95%, 50=81.02%, 100=0.04%
  cpu          : usr=0.80%, sys=3.70%, ctx=1335, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13002: Tue Apr 16 17:33:30 2024
  write: IOPS=665, BW=2661KiB/s (2725kB/s)(10.0MiB/3848msec)
    slat (usec): min=12, max=289, avg=46.08, stdev=25.86
    clat (usec): min=2749, max=62401, avg=23902.17, stdev=6056.02
     lat (usec): min=2792, max=62486, avg=23949.56, stdev=6062.70
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[17171], 10.00th=[18482], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22676], 60.00th=[23462],
     | 70.00th=[24511], 80.00th=[27132], 90.00th=[31327], 95.00th=[36963],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[55837], 99.95th=[55837],
     | 99.99th=[62653]
   bw (  KiB/s): min= 2064, max= 3128, per=24.95%, avg=2640.29, stdev=332.11, samples=7
   iops        : min=  516, max=  782, avg=660.00, stdev=83.01, samples=7
  lat (msec)   : 4=0.12%, 10=0.47%, 20=18.71%, 50=80.51%, 100=0.20%
  cpu          : usr=0.88%, sys=3.74%, ctx=1256, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13003: Tue Apr 16 17:33:30 2024
  write: IOPS=662, BW=2648KiB/s (2712kB/s)(10.0MiB/3867msec)
    slat (usec): min=12, max=1830, avg=45.69, stdev=43.38
    clat (usec): min=10499, max=60969, avg=24017.95, stdev=6049.43
     lat (usec): min=10555, max=60992, avg=24064.84, stdev=6055.97
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[17171], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22676], 60.00th=[23462],
     | 70.00th=[24511], 80.00th=[27132], 90.00th=[31065], 95.00th=[36963],
     | 99.00th=[44827], 99.50th=[45876], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min= 2096, max= 3072, per=24.88%, avg=2633.43, stdev=317.47, samples=7
   iops        : min=  524, max=  768, avg=658.29, stdev=79.33, samples=7
  lat (msec)   : 20=18.48%, 50=81.21%, 100=0.31%
  cpu          : usr=1.03%, sys=3.49%, ctx=1140, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.3MiB/s (10.8MB/s), 2646KiB/s-2661KiB/s (2709kB/s-2725kB/s), io=40.0MiB (41.9MB), run=3848-3870msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=13016: Tue Apr 16 17:33:34 2024
   read: IOPS=448, BW=1796KiB/s (1839kB/s)(5048KiB/2811msec)
    slat (usec): min=717, max=2602, avg=1187.89, stdev=248.83
    clat (usec): min=10418, max=25139, avg=16679.16, stdev=2216.29
     lat (usec): min=11558, max=26684, avg=17868.85, stdev=2231.78
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13042], 10.00th=[13829], 20.00th=[14615],
     | 30.00th=[15533], 40.00th=[16057], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[21890], 99.50th=[22938], 99.90th=[24773], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1600, max= 1848, per=25.01%, avg=1785.60, stdev=104.83, samples=5
   iops        : min=  400, max=  462, avg=446.40, stdev=26.21, samples=5
  write: IOPS=461, BW=1847KiB/s (1891kB/s)(5192KiB/2811msec)
    slat (usec): min=13, max=1055, avg=41.89, stdev=38.20
    clat (usec): min=11670, max=25366, avg=16961.61, stdev=1845.75
     lat (usec): min=11738, max=25504, avg=17004.50, stdev=1846.03
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14091], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[22152], 99.50th=[22676], 99.90th=[23987], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 1704, max= 1952, per=25.17%, avg=1870.40, stdev=99.82, samples=5
   iops        : min=  426, max=  488, avg=467.60, stdev=24.96, samples=5
  lat (msec)   : 20=94.18%, 50=5.82%
  cpu          : usr=0.57%, sys=7.54%, ctx=2216, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13017: Tue Apr 16 17:33:34 2024
   read: IOPS=442, BW=1771KiB/s (1814kB/s)(4976KiB/2809msec)
    slat (usec): min=678, max=2683, avg=1190.30, stdev=256.99
    clat (usec): min=2921, max=24928, avg=16580.81, stdev=2216.56
     lat (usec): min=4223, max=25928, avg=17772.85, stdev=2229.91
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13042], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21365], 99.50th=[22414], 99.90th=[24249], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1560, max= 1872, per=24.77%, avg=1768.00, stdev=124.58, samples=5
   iops        : min=  390, max=  468, avg=442.00, stdev=31.14, samples=5
  write: IOPS=468, BW=1874KiB/s (1919kB/s)(5264KiB/2809msec)
    slat (usec): min=12, max=154, avg=39.66, stdev=16.48
    clat (usec): min=5160, max=28693, avg=17097.71, stdev=1933.49
     lat (usec): min=5208, max=28810, avg=17138.42, stdev=1933.90
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14222], 10.00th=[14746], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22152], 99.50th=[23987], 99.90th=[26346], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1800, max= 2008, per=25.47%, avg=1892.80, stdev=90.08, samples=5
   iops        : min=  450, max=  502, avg=473.20, stdev=22.52, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=93.91%, 50=5.98%
  cpu          : usr=1.00%, sys=7.05%, ctx=2181, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13018: Tue Apr 16 17:33:34 2024
   read: IOPS=446, BW=1788KiB/s (1831kB/s)(4984KiB/2788msec)
    slat (usec): min=13, max=3120, avg=1180.93, stdev=262.97
    clat (usec): min=2025, max=23588, avg=16605.47, stdev=2342.53
     lat (usec): min=2050, max=25170, avg=17788.05, stdev=2370.17
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13173], 10.00th=[13829], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[21890], 99.50th=[22152], 99.90th=[22938], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1712, max= 1944, per=25.26%, avg=1803.20, stdev=86.64, samples=5
   iops        : min=  428, max=  486, avg=450.80, stdev=21.66, samples=5
  write: IOPS=471, BW=1885KiB/s (1930kB/s)(5256KiB/2788msec)
    slat (usec): min=13, max=1332, avg=42.23, stdev=44.29
    clat (usec): min=8623, max=31222, avg=16953.24, stdev=2002.04
     lat (usec): min=8647, max=31260, avg=16996.48, stdev=2006.88
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13960], 10.00th=[14615], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[22676], 99.50th=[25560], 99.90th=[27919], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1744, max= 1984, per=25.40%, avg=1888.00, stdev=106.88, samples=5
   iops        : min=  436, max=  496, avg=472.00, stdev=26.72, samples=5
  lat (msec)   : 4=0.12%, 10=0.23%, 20=93.44%, 50=6.21%
  cpu          : usr=1.15%, sys=6.96%, ctx=2153, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13019: Tue Apr 16 17:33:34 2024
   read: IOPS=450, BW=1800KiB/s (1843kB/s)(5060KiB/2811msec)
    slat (usec): min=674, max=2575, avg=1201.68, stdev=255.64
    clat (usec): min=2411, max=24960, avg=16564.80, stdev=2442.34
     lat (usec): min=3336, max=26704, avg=17768.07, stdev=2459.21
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13042], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22152], 99.50th=[22938], 99.90th=[24773], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1656, max= 1904, per=25.42%, avg=1814.40, stdev=95.90, samples=5
   iops        : min=  414, max=  476, avg=453.60, stdev=23.97, samples=5
  write: IOPS=460, BW=1843KiB/s (1887kB/s)(5180KiB/2811msec)
    slat (usec): min=13, max=1280, avg=41.78, stdev=51.01
    clat (usec): min=10705, max=25702, avg=17097.87, stdev=1982.76
     lat (usec): min=10738, max=25729, avg=17140.67, stdev=1983.86
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13960], 10.00th=[14615], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22414], 99.50th=[23462], 99.90th=[25560], 99.95th=[25822],
     | 99.99th=[25822]
   bw (  KiB/s): min= 1696, max= 1944, per=24.76%, avg=1840.00, stdev=94.15, samples=5
   iops        : min=  424, max=  486, avg=460.00, stdev=23.54, samples=5
  lat (msec)   : 4=0.12%, 10=0.08%, 20=92.38%, 50=7.42%
  cpu          : usr=1.07%, sys=6.87%, ctx=2184, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7139KiB/s (7310kB/s), 1771KiB/s-1800KiB/s (1814kB/s-1843kB/s), io=19.6MiB (20.5MB), run=2788-2811msec
  WRITE: bw=7432KiB/s (7611kB/s), 1843KiB/s-1885KiB/s (1887kB/s-1930kB/s), io=20.4MiB (21.4MB), run=2788-2811msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=13025: Tue Apr 16 17:33:35 2024
   read: IOPS=11, BW=449MiB/s (471MB/s)(40.0MiB/89msec)
    slat (nsec): min=5779.7k, max=5779.7k, avg=5779694.00, stdev= 0.00
    clat (nsec): min=78903k, max=78903k, avg=78902581.00, stdev= 0.00
     lat (nsec): min=84695k, max=84695k, avg=84695463.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[79168],  5.00th=[79168], 10.00th=[79168], 20.00th=[79168],
     | 30.00th=[79168], 40.00th=[79168], 50.00th=[79168], 60.00th=[79168],
     | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168],
     | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.82%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=40.0MiB (41.9MB), run=89-89msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout