== sanity test 398c: run fio to test AIO ================= 18:02:48 (1713477768)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0807576 s, 519 MB/s
osc.lustre-OST0000-osc-ffff88012bd97000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16273: Thu Apr 18 18:02:53 2024
  write: IOPS=716, BW=2866KiB/s (2935kB/s)(10.0MiB/3573msec)
    slat (usec): min=14, max=281, avg=46.84, stdev=25.19
    clat (usec): min=3669, max=47308, avg=22181.07, stdev=2840.49
     lat (usec): min=3691, max=47345, avg=22229.03, stdev=2841.31
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[18482], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[21627], 40.00th=[22152], 50.00th=[22676], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[23725], 90.00th=[24249], 95.00th=[24773],
     | 99.00th=[25822], 99.50th=[32113], 99.90th=[44827], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min= 2736, max= 3208, per=25.06%, avg=2869.71, stdev=164.32, samples=7
   iops        : min=  684, max=  802, avg=717.43, stdev=41.08, samples=7
  lat (msec)   : 4=0.08%, 10=0.94%, 20=9.88%, 50=89.10%
  cpu          : usr=0.90%, sys=3.50%, ctx=1640, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16274: Thu Apr 18 18:02:53 2024
  write: IOPS=716, BW=2864KiB/s (2933kB/s)(10.0MiB/3575msec)
    slat (usec): min=13, max=214, avg=43.90, stdev=24.17
    clat (usec): min=5934, max=37116, avg=22197.40, stdev=2282.67
     lat (usec): min=5993, max=37268, avg=22242.39, stdev=2285.03
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[18482], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[21627], 40.00th=[22152], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23200], 80.00th=[23725], 90.00th=[24249], 95.00th=[24773],
     | 99.00th=[25822], 99.50th=[26084], 99.90th=[34341], 99.95th=[34866],
     | 99.99th=[36963]
   bw (  KiB/s): min= 2746, max= 3088, per=24.98%, avg=2859.71, stdev=121.26, samples=7
   iops        : min=  686, max=  772, avg=714.86, stdev=30.39, samples=7
  lat (msec)   : 10=0.39%, 20=10.94%, 50=88.67%
  cpu          : usr=0.78%, sys=3.47%, ctx=1496, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16275: Thu Apr 18 18:02:53 2024
  write: IOPS=716, BW=2864KiB/s (2933kB/s)(10.0MiB/3575msec)
    slat (usec): min=13, max=313, avg=40.41, stdev=21.40
    clat (usec): min=2542, max=41908, avg=22198.83, stdev=2351.02
     lat (usec): min=2602, max=41931, avg=22240.32, stdev=2350.11
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[18744], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[21627], 40.00th=[22152], 50.00th=[22676], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[23725], 90.00th=[24249], 95.00th=[24773],
     | 99.00th=[25822], 99.50th=[26084], 99.90th=[32113], 99.95th=[39584],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2712, max= 3072, per=24.96%, avg=2857.43, stdev=128.10, samples=7
   iops        : min=  678, max=  768, avg=714.29, stdev=32.07, samples=7
  lat (msec)   : 4=0.04%, 10=0.47%, 20=10.51%, 50=88.98%
  cpu          : usr=0.50%, sys=3.64%, ctx=1158, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16276: Thu Apr 18 18:02:53 2024
  write: IOPS=715, BW=2863KiB/s (2931kB/s)(10.0MiB/3577msec)
    slat (usec): min=13, max=297, avg=47.15, stdev=24.21
    clat (usec): min=3557, max=36265, avg=22203.53, stdev=2346.71
     lat (usec): min=3619, max=36327, avg=22251.84, stdev=2347.26
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[18482], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[21627], 40.00th=[22152], 50.00th=[22676], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[23725], 90.00th=[24249], 95.00th=[24773],
     | 99.00th=[25822], 99.50th=[26084], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[36439]
   bw (  KiB/s): min= 2736, max= 3120, per=25.01%, avg=2864.00, stdev=137.25, samples=7
   iops        : min=  684, max=  780, avg=716.00, stdev=34.31, samples=7
  lat (msec)   : 4=0.04%, 10=0.35%, 20=10.59%, 50=89.02%
  cpu          : usr=0.84%, sys=3.75%, ctx=1807, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.2MiB/s (11.7MB/s), 2863KiB/s-2866KiB/s (2931kB/s-2935kB/s), io=40.0MiB (41.9MB), run=3573-3577msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16289: Thu Apr 18 18:02:58 2024
   read: IOPS=383, BW=1535KiB/s (1572kB/s)(5048KiB/3289msec)
    slat (usec): min=23, max=3286, avg=1350.01, stdev=299.56
    clat (usec): min=1652, max=38606, avg=19509.77, stdev=3156.54
     lat (usec): min=1778, max=40155, avg=20861.68, stdev=3230.45
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[15008], 10.00th=[16057], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18744], 50.00th=[19530], 60.00th=[20317],
     | 70.00th=[20841], 80.00th=[21890], 90.00th=[23462], 95.00th=[24511],
     | 99.00th=[26870], 99.50th=[28181], 99.90th=[35914], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1456, max= 1592, per=25.19%, avg=1533.33, stdev=52.56, samples=6
   iops        : min=  364, max=  398, avg=383.33, stdev=13.14, samples=6
  write: IOPS=394, BW=1579KiB/s (1616kB/s)(5192KiB/3289msec)
    slat (usec): min=13, max=1823, avg=49.35, stdev=55.23
    clat (usec): min=4781, max=35691, avg=20117.37, stdev=2439.52
     lat (usec): min=4837, max=35727, avg=20167.92, stdev=2439.57
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[16712], 10.00th=[17433], 20.00th=[18220],
     | 30.00th=[19006], 40.00th=[19530], 50.00th=[20055], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[21890], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[26346], 99.50th=[26870], 99.90th=[34341], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1413, max= 1664, per=25.11%, avg=1590.83, stdev=104.42, samples=6
   iops        : min=  353, max=  416, avg=397.67, stdev=26.19, samples=6
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.39%, 20=52.38%, 50=47.11%
  cpu          : usr=0.58%, sys=7.12%, ctx=2219, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16290: Thu Apr 18 18:02:58 2024
   read: IOPS=378, BW=1513KiB/s (1549kB/s)(4976KiB/3289msec)
    slat (usec): min=796, max=3314, avg=1361.71, stdev=285.82
    clat (usec): min=3718, max=40324, avg=19524.41, stdev=2965.92
     lat (usec): min=4907, max=41521, avg=20888.28, stdev=2989.34
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15270], 10.00th=[16057], 20.00th=[17171],
     | 30.00th=[17957], 40.00th=[18744], 50.00th=[19530], 60.00th=[20317],
     | 70.00th=[20841], 80.00th=[21627], 90.00th=[23200], 95.00th=[24249],
     | 99.00th=[26346], 99.50th=[30016], 99.90th=[39060], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1429, max= 1608, per=24.81%, avg=1509.67, stdev=76.00, samples=6
   iops        : min=  357, max=  402, avg=377.33, stdev=19.08, samples=6
  write: IOPS=400, BW=1600KiB/s (1639kB/s)(5264KiB/3289msec)
    slat (usec): min=13, max=231, avg=46.95, stdev=22.73
    clat (usec): min=12817, max=40281, avg=20031.68, stdev=2379.63
     lat (usec): min=12852, max=40316, avg=20079.93, stdev=2380.53
    clat percentiles (usec):
     |  1.00th=[15139],  5.00th=[16319], 10.00th=[17171], 20.00th=[18220],
     | 30.00th=[18744], 40.00th=[19268], 50.00th=[20055], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[21890], 90.00th=[22938], 95.00th=[23987],
     | 99.00th=[26346], 99.50th=[27132], 99.90th=[32900], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1508, max= 1728, per=25.53%, avg=1617.33, stdev=103.84, samples=6
   iops        : min=  377, max=  432, avg=404.33, stdev=25.96, samples=6
  lat (msec)   : 4=0.04%, 10=0.08%, 20=53.59%, 50=46.29%
  cpu          : usr=0.79%, sys=7.06%, ctx=2213, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16291: Thu Apr 18 18:02:58 2024
   read: IOPS=378, BW=1513KiB/s (1549kB/s)(4984KiB/3295msec)
    slat (usec): min=735, max=2962, avg=1366.64, stdev=280.79
    clat (usec): min=3691, max=31556, avg=19396.16, stdev=2786.80
     lat (usec): min=4697, max=33352, avg=20764.86, stdev=2817.93
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[15270], 10.00th=[16057], 20.00th=[17171],
     | 30.00th=[17957], 40.00th=[18744], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20841], 80.00th=[21627], 90.00th=[22676], 95.00th=[23987],
     | 99.00th=[25822], 99.50th=[26870], 99.90th=[30540], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 1405, max= 1688, per=24.96%, avg=1518.83, stdev=103.12, samples=6
   iops        : min=  351, max=  422, avg=379.67, stdev=25.84, samples=6
  write: IOPS=398, BW=1595KiB/s (1633kB/s)(5256KiB/3295msec)
    slat (usec): min=13, max=251, avg=47.27, stdev=21.95
    clat (usec): min=6019, max=31400, avg=20186.90, stdev=2277.79
     lat (usec): min=6052, max=31429, avg=20235.55, stdev=2277.74
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[16712], 10.00th=[17433], 20.00th=[18220],
     | 30.00th=[19006], 40.00th=[19530], 50.00th=[20055], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[21890], 90.00th=[22938], 95.00th=[23987],
     | 99.00th=[26346], 99.50th=[27395], 99.90th=[31327], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1516, max= 1720, per=25.23%, avg=1598.67, stdev=71.58, samples=6
   iops        : min=  379, max=  430, avg=399.67, stdev=17.90, samples=6
  lat (msec)   : 4=0.04%, 10=0.20%, 20=51.68%, 50=48.09%
  cpu          : usr=0.88%, sys=6.92%, ctx=2252, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16292: Thu Apr 18 18:02:58 2024
   read: IOPS=383, BW=1535KiB/s (1572kB/s)(5060KiB/3297msec)
    slat (usec): min=812, max=3193, avg=1367.66, stdev=282.95
    clat (usec): min=5682, max=29313, avg=19501.90, stdev=2727.91
     lat (usec): min=7978, max=30397, avg=20871.51, stdev=2753.28
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15270], 10.00th=[16188], 20.00th=[17171],
     | 30.00th=[17957], 40.00th=[18744], 50.00th=[19530], 60.00th=[20055],
     | 70.00th=[20841], 80.00th=[21627], 90.00th=[23200], 95.00th=[23987],
     | 99.00th=[26608], 99.50th=[27395], 99.90th=[28705], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1440, max= 1600, per=25.33%, avg=1541.33, stdev=61.07, samples=6
   iops        : min=  360, max=  400, avg=385.33, stdev=15.27, samples=6
  write: IOPS=392, BW=1571KiB/s (1609kB/s)(5180KiB/3297msec)
    slat (usec): min=15, max=1380, avg=49.64, stdev=52.68
    clat (usec): min=7986, max=28268, avg=20111.13, stdev=2297.48
     lat (usec): min=8064, max=28316, avg=20162.00, stdev=2299.98
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[16909], 10.00th=[17433], 20.00th=[18220],
     | 30.00th=[19006], 40.00th=[19530], 50.00th=[20055], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[21890], 90.00th=[23200], 95.00th=[23987],
     | 99.00th=[26346], 99.50th=[26870], 99.90th=[28181], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1480, max= 1696, per=24.77%, avg=1569.33, stdev=77.30, samples=6
   iops        : min=  370, max=  424, avg=392.33, stdev=19.33, samples=6
  lat (msec)   : 10=0.23%, 20=53.55%, 50=46.21%
  cpu          : usr=0.82%, sys=7.07%, ctx=2287, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6087KiB/s (6233kB/s), 1513KiB/s-1535KiB/s (1549kB/s-1572kB/s), io=19.6MiB (20.5MB), run=3289-3297msec
  WRITE: bw=6337KiB/s (6489kB/s), 1571KiB/s-1600KiB/s (1609kB/s-1639kB/s), io=20.4MiB (21.4MB), run=3289-3297msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16299: Thu Apr 18 18:02:59 2024
   read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec)
    slat (nsec): min=10507k, max=10507k, avg=10506731.00, stdev= 0.00
    clat (nsec): min=79074k, max=79074k, avg=79074184.00, stdev= 0.00
     lat (nsec): min=89686k, max=89686k, avg=89686300.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[79168],  5.00th=[79168], 10.00th=[79168], 20.00th=[79168],
     | 30.00th=[79168], 40.00th=[79168], 50.00th=[79168], 60.00th=[79168],
     | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168],
     | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.75%, ctx=11, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout