== sanity test 398c: run fio to test AIO ================= 07:12:11 (1713438731)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0891282 s, 471 MB/s
osc.lustre-OST0000-osc-ffff8800b602e800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16010: Thu Apr 18 07:12:15 2024
  write: IOPS=775, BW=3100KiB/s (3175kB/s)(10.0MiB/3303msec)
    slat (usec): min=11, max=172, avg=42.06, stdev=21.27
    clat (usec): min=3067, max=42530, avg=20521.35, stdev=2518.72
     lat (usec): min=3118, max=42571, avg=20564.42, stdev=2518.03
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[16581], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22938], 95.00th=[23462],
     | 99.00th=[26084], 99.50th=[28443], 99.90th=[35390], 99.95th=[38011],
     | 99.99th=[42730]
   bw (  KiB/s): min= 2904, max= 3360, per=25.16%, avg=3116.33, stdev=160.19, samples=6
   iops        : min=  726, max=  840, avg=779.00, stdev=40.13, samples=6
  lat (msec)   : 4=0.08%, 10=0.20%, 20=35.39%, 50=64.34%
  cpu          : usr=0.70%, sys=3.63%, ctx=1697, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16011: Thu Apr 18 07:12:15 2024
  write: IOPS=777, BW=3111KiB/s (3185kB/s)(10.0MiB/3292msec)
    slat (usec): min=12, max=205, avg=43.56, stdev=20.78
    clat (usec): min=4175, max=42825, avg=20457.00, stdev=2753.19
     lat (usec): min=4230, max=42877, avg=20501.64, stdev=2752.53
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[16057], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[20317], 50.00th=[20841], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[26084], 99.50th=[26608], 99.90th=[40109], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 2896, max= 3392, per=25.27%, avg=3129.67, stdev=172.39, samples=6
   iops        : min=  724, max=  848, avg=782.33, stdev=43.18, samples=6
  lat (msec)   : 10=0.94%, 20=34.49%, 50=64.57%
  cpu          : usr=0.91%, sys=3.65%, ctx=1772, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16012: Thu Apr 18 07:12:15 2024
  write: IOPS=774, BW=3098KiB/s (3173kB/s)(10.0MiB/3305msec)
    slat (usec): min=12, max=251, avg=44.68, stdev=23.49
    clat (usec): min=7339, max=40101, avg=20539.16, stdev=2519.70
     lat (usec): min=7396, max=40134, avg=20584.98, stdev=2519.36
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[16581], 10.00th=[17695], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22152], 90.00th=[22938], 95.00th=[23462],
     | 99.00th=[25822], 99.50th=[26608], 99.90th=[37487], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2920, max= 3352, per=25.14%, avg=3113.67, stdev=157.31, samples=6
   iops        : min=  730, max=  838, avg=778.33, stdev=39.42, samples=6
  lat (msec)   : 10=0.35%, 20=34.41%, 50=65.23%
  cpu          : usr=0.58%, sys=3.78%, ctx=1974, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16013: Thu Apr 18 07:12:15 2024
  write: IOPS=774, BW=3096KiB/s (3171kB/s)(10.0MiB/3307msec)
    slat (usec): min=11, max=220, avg=40.83, stdev=21.78
    clat (usec): min=1781, max=34599, avg=20551.79, stdev=2348.95
     lat (usec): min=1903, max=34650, avg=20593.65, stdev=2349.28
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[16581], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[25297], 99.50th=[26084], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[34341]
   bw (  KiB/s): min= 2912, max= 3344, per=25.13%, avg=3112.33, stdev=149.60, samples=6
   iops        : min=  728, max=  836, avg=778.00, stdev=37.48, samples=6
  lat (msec)   : 2=0.04%, 10=0.23%, 20=33.75%, 50=65.98%
  cpu          : usr=0.85%, sys=3.27%, ctx=1509, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.1MiB/s (12.7MB/s), 3096KiB/s-3111KiB/s (3171kB/s-3185kB/s), io=40.0MiB (41.9MB), run=3292-3307msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16026: Thu Apr 18 07:12:19 2024
   read: IOPS=427, BW=1711KiB/s (1752kB/s)(5048KiB/2951msec)
    slat (usec): min=32, max=3808, avg=1245.87, stdev=282.87
    clat (usec): min=2555, max=35491, avg=17506.49, stdev=2893.41
     lat (usec): min=2598, max=37319, avg=18754.29, stdev=2927.08
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[13566], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19792], 90.00th=[20841], 95.00th=[21627],
     | 99.00th=[23987], 99.50th=[25822], 99.90th=[34341], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1656, max= 1756, per=25.02%, avg=1693.60, stdev=37.27, samples=5
   iops        : min=  414, max=  439, avg=423.40, stdev= 9.32, samples=5
  write: IOPS=439, BW=1759KiB/s (1802kB/s)(5192KiB/2951msec)
    slat (usec): min=12, max=1670, avg=44.23, stdev=49.38
    clat (usec): min=6063, max=34290, avg=18009.53, stdev=2174.57
     lat (usec): min=6221, max=34320, avg=18054.83, stdev=2174.33
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[14746], 10.00th=[15664], 20.00th=[16581],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19530], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[22676], 99.50th=[23200], 99.90th=[33162], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1712, max= 1832, per=25.30%, avg=1783.20, stdev=49.35, samples=5
   iops        : min=  428, max=  458, avg=445.80, stdev=12.34, samples=5
  lat (msec)   : 4=0.23%, 10=0.70%, 20=83.67%, 50=15.39%
  cpu          : usr=0.98%, sys=7.15%, ctx=2330, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16027: Thu Apr 18 07:12:19 2024
   read: IOPS=420, BW=1681KiB/s (1721kB/s)(4976KiB/2960msec)
    slat (usec): min=703, max=2467, avg=1248.17, stdev=258.27
    clat (usec): min=3022, max=28737, avg=17369.10, stdev=2490.01
     lat (usec): min=4305, max=30052, avg=18619.05, stdev=2501.50
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13566], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16712], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20317], 95.00th=[21365],
     | 99.00th=[23462], 99.50th=[24249], 99.90th=[27395], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1544, max= 1728, per=24.61%, avg=1666.40, stdev=71.64, samples=5
   iops        : min=  386, max=  432, avg=416.60, stdev=17.91, samples=5
  write: IOPS=444, BW=1778KiB/s (1821kB/s)(5264KiB/2960msec)
    slat (usec): min=12, max=550, avg=43.60, stdev=24.47
    clat (usec): min=10882, max=33426, avg=18168.25, stdev=2128.08
     lat (usec): min=10910, max=33497, avg=18212.87, stdev=2127.19
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14877], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[19792], 90.00th=[20841], 95.00th=[21627],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[30016], 99.95th=[33424],
     | 99.99th=[33424]
   bw (  KiB/s): min= 1704, max= 1868, per=25.46%, avg=1794.40, stdev=60.37, samples=5
   iops        : min=  426, max=  467, avg=448.60, stdev=15.09, samples=5
  lat (msec)   : 4=0.04%, 10=0.12%, 20=84.02%, 50=15.82%
  cpu          : usr=0.61%, sys=7.27%, ctx=2408, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16028: Thu Apr 18 07:12:19 2024
   read: IOPS=420, BW=1682KiB/s (1722kB/s)(4984KiB/2963msec)
    slat (usec): min=727, max=4963, avg=1250.76, stdev=271.54
    clat (usec): min=2581, max=24864, avg=17476.81, stdev=2420.84
     lat (usec): min=3985, max=25915, avg=18729.36, stdev=2421.66
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13698], 10.00th=[14484], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20579], 95.00th=[21103],
     | 99.00th=[22938], 99.50th=[23200], 99.90th=[24511], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 1472, max= 1852, per=24.78%, avg=1677.60, stdev=140.67, samples=5
   iops        : min=  368, max=  463, avg=419.40, stdev=35.17, samples=5
  write: IOPS=443, BW=1774KiB/s (1816kB/s)(5256KiB/2963msec)
    slat (usec): min=13, max=1293, avg=46.01, stdev=59.61
    clat (usec): min=10729, max=28867, avg=18075.56, stdev=1968.72
     lat (usec): min=10809, max=28900, avg=18122.67, stdev=1967.03
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[15008], 10.00th=[15795], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19530], 90.00th=[20317], 95.00th=[21365],
     | 99.00th=[22676], 99.50th=[23200], 99.90th=[28705], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1660, max= 1912, per=25.23%, avg=1778.40, stdev=101.02, samples=5
   iops        : min=  415, max=  478, avg=444.60, stdev=25.25, samples=5
  lat (msec)   : 4=0.04%, 10=0.04%, 20=84.88%, 50=15.04%
  cpu          : usr=0.78%, sys=6.99%, ctx=2398, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16029: Thu Apr 18 07:12:19 2024
   read: IOPS=426, BW=1707KiB/s (1748kB/s)(5060KiB/2964msec)
    slat (usec): min=758, max=3140, avg=1254.88, stdev=258.75
    clat (usec): min=2289, max=25229, avg=17517.90, stdev=2458.61
     lat (usec): min=3287, max=26657, avg=18774.71, stdev=2473.67
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13698], 10.00th=[14484], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20579], 95.00th=[21365],
     | 99.00th=[23200], 99.50th=[23987], 99.90th=[25297], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 1656, max= 1744, per=25.11%, avg=1700.00, stdev=36.88, samples=5
   iops        : min=  414, max=  436, avg=425.00, stdev= 9.22, samples=5
  write: IOPS=436, BW=1748KiB/s (1790kB/s)(5180KiB/2964msec)
    slat (usec): min=12, max=783, avg=44.11, stdev=30.31
    clat (usec): min=11164, max=27587, avg=18044.24, stdev=1969.57
     lat (usec): min=11236, max=27606, avg=18089.45, stdev=1969.03
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14746], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19530], 90.00th=[20579], 95.00th=[21365],
     | 99.00th=[22676], 99.50th=[23987], 99.90th=[27657], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1672, max= 1820, per=24.80%, avg=1748.00, stdev=59.60, samples=5
   iops        : min=  418, max=  455, avg=437.00, stdev=14.90, samples=5
  lat (msec)   : 4=0.04%, 10=0.04%, 20=84.26%, 50=15.66%
  cpu          : usr=0.71%, sys=7.09%, ctx=2481, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6771KiB/s (6933kB/s), 1681KiB/s-1711KiB/s (1721kB/s-1752kB/s), io=19.6MiB (20.5MB), run=2951-2964msec
  WRITE: bw=7049KiB/s (7218kB/s), 1748KiB/s-1778KiB/s (1790kB/s-1821kB/s), io=20.4MiB (21.4MB), run=2951-2964msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16035: Thu Apr 18 07:12:21 2024
   read: IOPS=12, BW=513MiB/s (538MB/s)(40.0MiB/78msec)
    slat (nsec): min=6115.6k, max=6115.6k, avg=6115586.00, stdev= 0.00
    clat (nsec): min=67356k, max=67356k, avg=67355861.00, stdev= 0.00
     lat (nsec): min=73485k, max=73485k, avg=73485335.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[67634],  5.00th=[67634], 10.00th=[67634], 20.00th=[67634],
     | 30.00th=[67634], 40.00th=[67634], 50.00th=[67634], 60.00th=[67634],
     | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634],
     | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[67634]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.79%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=513MiB/s (538MB/s), 513MiB/s-513MiB/s (538MB/s-538MB/s), io=40.0MiB (41.9MB), run=78-78msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout