== sanity test 398c: run fio to test AIO ================= 23:23:29 (1713410609)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.118661 s, 353 MB/s
osc.lustre-OST0000-osc-ffff8800923c0800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30906: Wed Apr 17 23:23:48 2024
  write: IOPS=142, BW=571KiB/s (585kB/s)(10.0MiB/17930msec)
    slat (usec): min=12, max=307, avg=43.25, stdev=23.84
    clat (msec): min=43, max=225, avg=111.56, stdev=13.28
     lat (msec): min=43, max=225, avg=111.60, stdev=13.28
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[   94], 10.00th=[   99], 20.00th=[  106],
     | 30.00th=[  107], 40.00th=[  108], 50.00th=[  110], 60.00th=[  111],
     | 70.00th=[  116], 80.00th=[  120], 90.00th=[  128], 95.00th=[  133],
     | 99.00th=[  150], 99.50th=[  159], 99.90th=[  205], 99.95th=[  215],
     | 99.99th=[  226]
   bw (  KiB/s): min=  520, max=  648, per=25.05%, avg=571.89, stdev=28.37, samples=35
   iops        : min=  130, max=  162, avg=142.86, stdev= 7.02, samples=35
  lat (msec)   : 50=0.04%, 100=12.15%, 250=87.81%
  cpu          : usr=0.10%, sys=0.78%, ctx=1487, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30907: Wed Apr 17 23:23:48 2024
  write: IOPS=142, BW=571KiB/s (585kB/s)(10.0MiB/17919msec)
    slat (usec): min=11, max=271, avg=44.30, stdev=27.02
    clat (msec): min=19, max=236, avg=111.49, stdev=13.73
     lat (msec): min=20, max=236, avg=111.54, stdev=13.73
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[   94], 10.00th=[  100], 20.00th=[  106],
     | 30.00th=[  107], 40.00th=[  109], 50.00th=[  110], 60.00th=[  112],
     | 70.00th=[  116], 80.00th=[  120], 90.00th=[  128], 95.00th=[  136],
     | 99.00th=[  150], 99.50th=[  161], 99.90th=[  215], 99.95th=[  215],
     | 99.99th=[  236]
   bw (  KiB/s): min=  520, max=  664, per=25.04%, avg=571.66, stdev=31.55, samples=35
   iops        : min=  130, max=  166, avg=142.91, stdev= 7.89, samples=35
  lat (msec)   : 20=0.04%, 50=0.20%, 100=11.95%, 250=87.81%
  cpu          : usr=0.08%, sys=0.80%, ctx=1726, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30908: Wed Apr 17 23:23:48 2024
  write: IOPS=143, BW=572KiB/s (586kB/s)(10.0MiB/17901msec)
    slat (usec): min=12, max=281, avg=45.58, stdev=27.47
    clat (msec): min=23, max=207, avg=111.37, stdev=13.20
     lat (msec): min=23, max=207, avg=111.42, stdev=13.20
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[   94], 10.00th=[   99], 20.00th=[  105],
     | 30.00th=[  107], 40.00th=[  109], 50.00th=[  110], 60.00th=[  111],
     | 70.00th=[  116], 80.00th=[  121], 90.00th=[  128], 95.00th=[  134],
     | 99.00th=[  148], 99.50th=[  159], 99.90th=[  197], 99.95th=[  197],
     | 99.99th=[  209]
   bw (  KiB/s): min=  520, max=  688, per=25.05%, avg=571.89, stdev=36.71, samples=35
   iops        : min=  130, max=  172, avg=142.91, stdev= 9.21, samples=35
  lat (msec)   : 50=0.16%, 100=12.54%, 250=87.30%
  cpu          : usr=0.07%, sys=0.84%, ctx=1677, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30909: Wed Apr 17 23:23:48 2024
  write: IOPS=142, BW=571KiB/s (584kB/s)(10.0MiB/17940msec)
    slat (usec): min=12, max=283, avg=44.15, stdev=26.35
    clat (msec): min=19, max=168, avg=111.64, stdev=12.20
     lat (msec): min=19, max=168, avg=111.68, stdev=12.20
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[   95], 10.00th=[  100], 20.00th=[  106],
     | 30.00th=[  107], 40.00th=[  109], 50.00th=[  110], 60.00th=[  112],
     | 70.00th=[  116], 80.00th=[  121], 90.00th=[  128], 95.00th=[  133],
     | 99.00th=[  148], 99.50th=[  150], 99.90th=[  169], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  510, max=  648, per=24.93%, avg=569.20, stdev=29.53, samples=35
   iops        : min=  127, max=  162, avg=142.26, stdev= 7.42, samples=35
  lat (msec)   : 20=0.04%, 50=0.12%, 100=11.17%, 250=88.67%
  cpu          : usr=0.14%, sys=0.75%, ctx=1690, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=2283KiB/s (2338kB/s), 571KiB/s-572KiB/s (584kB/s-586kB/s), io=40.0MiB (41.9MB), run=17901-17940msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30933: Wed Apr 17 23:24:01 2024
   read: IOPS=112, BW=451KiB/s (462kB/s)(5048KiB/11195msec)
    slat (usec): min=25, max=2791, avg=1191.84, stdev=269.03
    clat (msec): min=4, max=145, avg=65.02, stdev=11.71
     lat (msec): min=4, max=145, avg=66.22, stdev=11.72
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   50], 10.00th=[   55], 20.00th=[   58],
     | 30.00th=[   59], 40.00th=[   62], 50.00th=[   66], 60.00th=[   68],
     | 70.00th=[   70], 80.00th=[   72], 90.00th=[   78], 95.00th=[   81],
     | 99.00th=[   99], 99.50th=[  136], 99.90th=[  146], 99.95th=[  146],
     | 99.99th=[  146]
   bw (  KiB/s): min=  328, max=  528, per=25.14%, avg=450.00, stdev=46.27, samples=22
   iops        : min=   82, max=  132, avg=112.36, stdev=11.57, samples=22
  write: IOPS=115, BW=464KiB/s (475kB/s)(5192KiB/11195msec)
    slat (usec): min=12, max=882, avg=45.01, stdev=32.90
    clat (msec): min=31, max=157, avg=73.48, stdev=11.60
     lat (msec): min=31, max=157, avg=73.53, stdev=11.60
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   58], 10.00th=[   61], 20.00th=[   66],
     | 30.00th=[   69], 40.00th=[   70], 50.00th=[   71], 60.00th=[   73],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   88], 95.00th=[   94],
     | 99.00th=[  113], 99.50th=[  125], 99.90th=[  157], 99.95th=[  159],
     | 99.99th=[  159]
   bw (  KiB/s): min=  392, max=  552, per=24.94%, avg=464.91, stdev=37.80, samples=22
   iops        : min=   98, max=  138, avg=116.09, stdev= 9.46, samples=22
  lat (msec)   : 10=0.12%, 50=2.81%, 100=95.51%, 250=1.56%
  cpu          : usr=0.33%, sys=1.94%, ctx=2428, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30934: Wed Apr 17 23:24:01 2024
   read: IOPS=111, BW=444KiB/s (455kB/s)(4976KiB/11206msec)
    slat (usec): min=48, max=2802, avg=1201.57, stdev=264.99
    clat (msec): min=12, max=149, avg=64.79, stdev=10.42
     lat (msec): min=14, max=149, avg=65.99, stdev=10.40
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   50], 10.00th=[   55], 20.00th=[   58],
     | 30.00th=[   59], 40.00th=[   62], 50.00th=[   66], 60.00th=[   68],
     | 70.00th=[   70], 80.00th=[   72], 90.00th=[   75], 95.00th=[   80],
     | 99.00th=[   88], 99.50th=[   91], 99.90th=[  148], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=  320, max=  528, per=24.69%, avg=442.00, stdev=54.01, samples=22
   iops        : min=   80, max=  132, avg=110.36, stdev=13.48, samples=22
  write: IOPS=117, BW=470KiB/s (481kB/s)(5264KiB/11206msec)
    slat (usec): min=13, max=228, avg=44.29, stdev=22.64
    clat (msec): min=28, max=144, avg=73.75, stdev=11.87
     lat (msec): min=28, max=144, avg=73.80, stdev=11.87
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   59], 10.00th=[   62], 20.00th=[   67],
     | 30.00th=[   69], 40.00th=[   70], 50.00th=[   72], 60.00th=[   73],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   87], 95.00th=[   94],
     | 99.00th=[  113], 99.50th=[  140], 99.90th=[  146], 99.95th=[  146],
     | 99.99th=[  146]
   bw (  KiB/s): min=  360, max=  552, per=25.09%, avg=467.73, stdev=42.93, samples=22
   iops        : min=   90, max=  138, avg=116.82, stdev=10.71, samples=22
  lat (msec)   : 20=0.08%, 50=2.89%, 100=95.12%, 250=1.91%
  cpu          : usr=0.29%, sys=1.89%, ctx=2535, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30935: Wed Apr 17 23:24:01 2024
   read: IOPS=111, BW=445KiB/s (456kB/s)(4984KiB/11196msec)
    slat (usec): min=28, max=2950, avg=1191.70, stdev=273.77
    clat (usec): min=1413, max=145873, avg=64991.43, stdev=10556.60
     lat (msec): min=2, max=145, avg=66.18, stdev=10.55
    clat percentiles (msec):
     |  1.00th=[   45],  5.00th=[   52], 10.00th=[   56], 20.00th=[   58],
     | 30.00th=[   59], 40.00th=[   62], 50.00th=[   65], 60.00th=[   68],
     | 70.00th=[   70], 80.00th=[   72], 90.00th=[   75], 95.00th=[   81],
     | 99.00th=[   91], 99.50th=[  123], 99.90th=[  144], 99.95th=[  146],
     | 99.99th=[  146]
   bw (  KiB/s): min=  320, max=  520, per=24.91%, avg=445.82, stdev=56.42, samples=22
   iops        : min=   80, max=  130, avg=111.45, stdev=14.10, samples=22
  write: IOPS=117, BW=469KiB/s (481kB/s)(5256KiB/11196msec)
    slat (usec): min=13, max=312, avg=44.12, stdev=23.01
    clat (msec): min=27, max=147, avg=73.46, stdev=11.95
     lat (msec): min=27, max=147, avg=73.50, stdev=11.95
    clat percentiles (msec):
     |  1.00th=[   54],  5.00th=[   58], 10.00th=[   61], 20.00th=[   67],
     | 30.00th=[   69], 40.00th=[   70], 50.00th=[   71], 60.00th=[   73],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   88], 95.00th=[   94],
     | 99.00th=[  114], 99.50th=[  140], 99.90th=[  142], 99.95th=[  148],
     | 99.99th=[  148]
   bw (  KiB/s): min=  384, max=  552, per=25.07%, avg=467.27, stdev=49.42, samples=22
   iops        : min=   96, max=  138, avg=116.82, stdev=12.35, samples=22
  lat (msec)   : 2=0.04%, 4=0.04%, 20=0.04%, 50=2.23%, 100=95.82%
  lat (msec)   : 250=1.84%
  cpu          : usr=0.18%, sys=2.02%, ctx=2590, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30936: Wed Apr 17 23:24:01 2024
   read: IOPS=112, BW=452KiB/s (463kB/s)(5060KiB/11196msec)
    slat (usec): min=27, max=2599, avg=1194.80, stdev=260.94
    clat (msec): min=13, max=148, avg=65.31, stdev=11.22
     lat (msec): min=14, max=148, avg=66.50, stdev=11.18
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   51], 10.00th=[   55], 20.00th=[   58],
     | 30.00th=[   59], 40.00th=[   62], 50.00th=[   66], 60.00th=[   69],
     | 70.00th=[   70], 80.00th=[   72], 90.00th=[   78], 95.00th=[   81],
     | 99.00th=[   92], 99.50th=[  146], 99.90th=[  148], 99.95th=[  148],
     | 99.99th=[  148]
   bw (  KiB/s): min=  304, max=  568, per=25.25%, avg=451.91, stdev=65.99, samples=22
   iops        : min=   76, max=  142, avg=112.91, stdev=16.56, samples=22
  write: IOPS=115, BW=463KiB/s (474kB/s)(5180KiB/11196msec)
    slat (usec): min=12, max=289, avg=44.37, stdev=22.88
    clat (msec): min=27, max=147, avg=73.24, stdev=11.79
     lat (msec): min=27, max=147, avg=73.29, stdev=11.79
    clat percentiles (msec):
     |  1.00th=[   53],  5.00th=[   58], 10.00th=[   61], 20.00th=[   66],
     | 30.00th=[   69], 40.00th=[   70], 50.00th=[   71], 60.00th=[   73],
     | 70.00th=[   77], 80.00th=[   82], 90.00th=[   87], 95.00th=[   93],
     | 99.00th=[  115], 99.50th=[  140], 99.90th=[  144], 99.95th=[  148],
     | 99.99th=[  148]
   bw (  KiB/s): min=  384, max=  528, per=24.79%, avg=462.05, stdev=40.53, samples=22
   iops        : min=   96, max=  132, avg=115.45, stdev=10.07, samples=22
  lat (msec)   : 20=0.08%, 50=2.50%, 100=95.70%, 250=1.72%
  cpu          : usr=0.31%, sys=1.92%, ctx=2524, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1791KiB/s (1834kB/s), 444KiB/s-452KiB/s (455kB/s-463kB/s), io=19.6MiB (20.5MB), run=11195-11206msec
  WRITE: bw=1864KiB/s (1909kB/s), 463KiB/s-470KiB/s (474kB/s-481kB/s), io=20.4MiB (21.4MB), run=11195-11206msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30949: Wed Apr 17 23:24:02 2024
   read: IOPS=6, BW=241MiB/s (253MB/s)(40.0MiB/166msec)
    slat (nsec): min=6136.5k, max=6136.5k, avg=6136536.00, stdev= 0.00
    clat (nsec): min=159332k, max=159332k, avg=159332190.00, stdev= 0.00
     lat (nsec): min=165477k, max=165477k, avg=165477182.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  159],  5.00th=[  159], 10.00th=[  159], 20.00th=[  159],
     | 30.00th=[  159], 40.00th=[  159], 50.00th=[  159], 60.00th=[  159],
     | 70.00th=[  159], 80.00th=[  159], 90.00th=[  159], 95.00th=[  159],
     | 99.00th=[  159], 99.50th=[  159], 99.90th=[  159], 99.95th=[  159],
     | 99.99th=[  159]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.03%, ctx=16, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=241MiB/s (253MB/s), 241MiB/s-241MiB/s (253MB/s-253MB/s), io=40.0MiB (41.9MB), run=166-166msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout