== sanity test 398c: run fio to test AIO ================= 18:27:41 (1713392861)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.127274 s, 330 MB/s
osc.lustre-OST0000-osc-ffff88012b70f800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=21870: Wed Apr 17 18:28:09 2024
  write: IOPS=97, BW=391KiB/s (400kB/s)(10.0MiB/26205msec)
    slat (usec): min=11, max=292, avg=48.72, stdev=24.51
    clat (msec): min=61, max=261, avg=163.69, stdev=29.62
     lat (msec): min=61, max=261, avg=163.74, stdev=29.62
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[  114], 10.00th=[  130], 20.00th=[  142],
     | 30.00th=[  150], 40.00th=[  157], 50.00th=[  165], 60.00th=[  171],
     | 70.00th=[  178], 80.00th=[  188], 90.00th=[  201], 95.00th=[  213],
     | 99.00th=[  234], 99.50th=[  243], 99.90th=[  251], 99.95th=[  259],
     | 99.99th=[  262]
   bw (  KiB/s): min=  280, max=  592, per=24.96%, avg=390.10, stdev=50.69, samples=52
   iops        : min=   70, max=  148, avg=97.48, stdev=12.69, samples=52
  lat (msec)   : 100=2.66%, 250=97.23%, 500=0.12%
  cpu          : usr=0.14%, sys=0.57%, ctx=2000, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21871: Wed Apr 17 18:28:09 2024
  write: IOPS=97, BW=391KiB/s (400kB/s)(10.0MiB/26196msec)
    slat (usec): min=12, max=262, avg=48.36, stdev=23.48
    clat (msec): min=26, max=286, avg=163.61, stdev=29.65
     lat (msec): min=26, max=286, avg=163.66, stdev=29.65
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[  115], 10.00th=[  129], 20.00th=[  142],
     | 30.00th=[  150], 40.00th=[  157], 50.00th=[  165], 60.00th=[  171],
     | 70.00th=[  180], 80.00th=[  188], 90.00th=[  201], 95.00th=[  211],
     | 99.00th=[  232], 99.50th=[  239], 99.90th=[  279], 99.95th=[  288],
     | 99.99th=[  288]
   bw (  KiB/s): min=  312, max=  544, per=24.95%, avg=389.96, stdev=47.51, samples=52
   iops        : min=   78, max=  136, avg=97.46, stdev=11.91, samples=52
  lat (msec)   : 50=0.08%, 100=2.19%, 250=97.54%, 500=0.20%
  cpu          : usr=0.14%, sys=0.59%, ctx=1937, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21872: Wed Apr 17 18:28:09 2024
  write: IOPS=97, BW=391KiB/s (401kB/s)(10.0MiB/26156msec)
    slat (usec): min=12, max=231, avg=49.68, stdev=24.40
    clat (msec): min=19, max=349, avg=163.33, stdev=31.16
     lat (msec): min=19, max=349, avg=163.38, stdev=31.16
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[  111], 10.00th=[  127], 20.00th=[  140],
     | 30.00th=[  148], 40.00th=[  157], 50.00th=[  165], 60.00th=[  171],
     | 70.00th=[  180], 80.00th=[  188], 90.00th=[  201], 95.00th=[  211],
     | 99.00th=[  239], 99.50th=[  255], 99.90th=[  317], 99.95th=[  317],
     | 99.99th=[  351]
   bw (  KiB/s): min=  320, max=  608, per=25.05%, avg=391.48, stdev=55.25, samples=52
   iops        : min=   80, max=  152, avg=97.83, stdev=13.83, samples=52
  lat (msec)   : 20=0.04%, 50=0.08%, 100=2.97%, 250=96.41%, 500=0.51%
  cpu          : usr=0.13%, sys=0.62%, ctx=2042, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21873: Wed Apr 17 18:28:09 2024
  write: IOPS=97, BW=391KiB/s (400kB/s)(10.0MiB/26186msec)
    slat (usec): min=12, max=218, avg=49.57, stdev=24.53
    clat (msec): min=61, max=297, avg=163.53, stdev=29.51
     lat (msec): min=61, max=297, avg=163.58, stdev=29.51
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[  115], 10.00th=[  128], 20.00th=[  142],
     | 30.00th=[  148], 40.00th=[  157], 50.00th=[  165], 60.00th=[  171],
     | 70.00th=[  180], 80.00th=[  188], 90.00th=[  201], 95.00th=[  213],
     | 99.00th=[  232], 99.50th=[  239], 99.90th=[  259], 99.95th=[  262],
     | 99.99th=[  300]
   bw (  KiB/s): min=  304, max=  552, per=25.00%, avg=390.73, stdev=48.21, samples=52
   iops        : min=   76, max=  138, avg=97.65, stdev=12.08, samples=52
  lat (msec)   : 100=2.34%, 250=97.46%, 500=0.20%
  cpu          : usr=0.14%, sys=0.61%, ctx=2003, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1563KiB/s (1601kB/s), 391KiB/s-391KiB/s (400kB/s-401kB/s), io=40.0MiB (41.9MB), run=26156-26205msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21904: Wed Apr 17 18:28:25 2024
   read: IOPS=83, BW=334KiB/s (343kB/s)(5048KiB/15092msec)
    slat (usec): min=449, max=2837, avg=737.09, stdev=200.63
    clat (usec): min=672, max=112229, avg=62589.39, stdev=11854.91
     lat (usec): min=1307, max=113833, avg=63328.37, stdev=11897.59
    clat percentiles (msec):
     |  1.00th=[   16],  5.00th=[   47], 10.00th=[   51], 20.00th=[   54],
     | 30.00th=[   57], 40.00th=[   61], 50.00th=[   63], 60.00th=[   65],
     | 70.00th=[   67], 80.00th=[   71], 90.00th=[   77], 95.00th=[   81],
     | 99.00th=[   96], 99.50th=[  106], 99.90th=[  111], 99.95th=[  113],
     | 99.99th=[  113]
   bw (  KiB/s): min=  215, max=  424, per=25.21%, avg=333.53, stdev=51.55, samples=30
   iops        : min=   53, max=  106, avg=83.33, stdev=12.92, samples=30
  write: IOPS=86, BW=344KiB/s (352kB/s)(5192KiB/15092msec)
    slat (usec): min=13, max=747, avg=47.74, stdev=33.77
    clat (msec): min=53, max=216, avg=124.14, stdev=28.89
     lat (msec): min=53, max=216, avg=124.19, stdev=28.89
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   81], 10.00th=[   89], 20.00th=[  100],
     | 30.00th=[  107], 40.00th=[  115], 50.00th=[  124], 60.00th=[  131],
     | 70.00th=[  140], 80.00th=[  150], 90.00th=[  161], 95.00th=[  176],
     | 99.00th=[  201], 99.50th=[  205], 99.90th=[  213], 99.95th=[  218],
     | 99.99th=[  218]
   bw (  KiB/s): min=  272, max=  432, per=24.82%, avg=342.07, stdev=38.18, samples=30
   iops        : min=   68, max=  108, avg=85.47, stdev= 9.53, samples=30
  lat (usec)   : 750=0.04%
  lat (msec)   : 2=0.04%, 20=0.51%, 50=3.36%, 100=55.74%, 250=40.31%
  cpu          : usr=0.27%, sys=1.46%, ctx=2870, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21905: Wed Apr 17 18:28:25 2024
   read: IOPS=82, BW=329KiB/s (337kB/s)(4976KiB/15139msec)
    slat (usec): min=405, max=3081, avg=731.52, stdev=202.53
    clat (usec): min=1660, max=117546, avg=62976.93, stdev=11417.26
     lat (msec): min=2, max=118, avg=63.71, stdev=11.44
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   48], 10.00th=[   51], 20.00th=[   54],
     | 30.00th=[   57], 40.00th=[   61], 50.00th=[   64], 60.00th=[   66],
     | 70.00th=[   67], 80.00th=[   71], 90.00th=[   78], 95.00th=[   81],
     | 99.00th=[   93], 99.50th=[  111], 99.90th=[  117], 99.95th=[  118],
     | 99.99th=[  118]
   bw (  KiB/s): min=  232, max=  440, per=24.69%, avg=326.60, stdev=48.08, samples=30
   iops        : min=   58, max=  110, avg=81.60, stdev=12.04, samples=30
  write: IOPS=86, BW=348KiB/s (356kB/s)(5264KiB/15139msec)
    slat (usec): min=12, max=433, avg=46.67, stdev=24.15
    clat (msec): min=27, max=219, avg=123.59, stdev=28.29
     lat (msec): min=28, max=220, avg=123.63, stdev=28.29
    clat percentiles (msec):
     |  1.00th=[   68],  5.00th=[   79], 10.00th=[   88], 20.00th=[  100],
     | 30.00th=[  108], 40.00th=[  115], 50.00th=[  123], 60.00th=[  130],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  159], 95.00th=[  171],
     | 99.00th=[  194], 99.50th=[  199], 99.90th=[  218], 99.95th=[  220],
     | 99.99th=[  220]
   bw (  KiB/s): min=  272, max=  424, per=25.17%, avg=346.87, stdev=41.73, samples=30
   iops        : min=   68, max=  106, avg=86.67, stdev=10.43, samples=30
  lat (msec)   : 2=0.04%, 20=0.20%, 50=3.79%, 100=55.08%, 250=40.90%
  cpu          : usr=0.23%, sys=1.45%, ctx=2724, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21906: Wed Apr 17 18:28:25 2024
   read: IOPS=82, BW=329KiB/s (337kB/s)(4984KiB/15159msec)
    slat (usec): min=462, max=2653, avg=743.63, stdev=209.07
    clat (msec): min=9, max=113, avg=63.03, stdev=11.70
     lat (msec): min=10, max=115, avg=63.78, stdev=11.74
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   48], 10.00th=[   51], 20.00th=[   54],
     | 30.00th=[   57], 40.00th=[   61], 50.00th=[   63], 60.00th=[   66],
     | 70.00th=[   68], 80.00th=[   72], 90.00th=[   78], 95.00th=[   82],
     | 99.00th=[  100], 99.50th=[  110], 99.90th=[  114], 99.95th=[  114],
     | 99.99th=[  114]
   bw (  KiB/s): min=  232, max=  424, per=24.83%, avg=328.50, stdev=49.55, samples=30
   iops        : min=   58, max=  106, avg=82.10, stdev=12.40, samples=30
  write: IOPS=86, BW=347KiB/s (355kB/s)(5256KiB/15159msec)
    slat (usec): min=12, max=239, avg=47.35, stdev=24.54
    clat (msec): min=56, max=220, avg=123.82, stdev=28.29
     lat (msec): min=57, max=220, avg=123.87, stdev=28.29
    clat percentiles (msec):
     |  1.00th=[   68],  5.00th=[   80], 10.00th=[   88], 20.00th=[  100],
     | 30.00th=[  109], 40.00th=[  115], 50.00th=[  123], 60.00th=[  130],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  161], 95.00th=[  169],
     | 99.00th=[  199], 99.50th=[  203], 99.90th=[  218], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  256, max=  416, per=24.90%, avg=343.17, stdev=40.42, samples=30
   iops        : min=   64, max=  104, avg=85.77, stdev=10.10, samples=30
  lat (msec)   : 10=0.04%, 20=0.23%, 50=3.75%, 100=54.77%, 250=41.21%
  cpu          : usr=0.23%, sys=1.46%, ctx=3083, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21907: Wed Apr 17 18:28:25 2024
   read: IOPS=84, BW=336KiB/s (344kB/s)(5060KiB/15053msec)
    slat (usec): min=37, max=2292, avg=731.97, stdev=202.72
    clat (usec): min=1173, max=141917, avg=62846.97, stdev=12093.35
     lat (usec): min=1417, max=142741, avg=63580.68, stdev=12134.47
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   48], 10.00th=[   52], 20.00th=[   55],
     | 30.00th=[   57], 40.00th=[   61], 50.00th=[   63], 60.00th=[   66],
     | 70.00th=[   67], 80.00th=[   71], 90.00th=[   77], 95.00th=[   80],
     | 99.00th=[   96], 99.50th=[  115], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  232, max=  464, per=25.37%, avg=335.70, stdev=54.49, samples=30
   iops        : min=   58, max=  116, avg=83.90, stdev=13.60, samples=30
  write: IOPS=86, BW=344KiB/s (352kB/s)(5180KiB/15053msec)
    slat (usec): min=13, max=460, avg=48.58, stdev=26.82
    clat (msec): min=27, max=216, avg=123.69, stdev=28.65
     lat (msec): min=27, max=216, avg=123.74, stdev=28.65
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   80], 10.00th=[   88], 20.00th=[  100],
     | 30.00th=[  108], 40.00th=[  115], 50.00th=[  123], 60.00th=[  131],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  159], 95.00th=[  171],
     | 99.00th=[  197], 99.50th=[  203], 99.90th=[  211], 99.95th=[  218],
     | 99.99th=[  218]
   bw (  KiB/s): min=  272, max=  408, per=24.88%, avg=342.90, stdev=34.84, samples=30
   iops        : min=   68, max=  102, avg=85.70, stdev= 8.72, samples=30
  lat (msec)   : 2=0.12%, 4=0.08%, 20=0.23%, 50=3.48%, 100=55.82%
  lat (msec)   : 250=40.27%
  cpu          : usr=0.21%, sys=1.49%, ctx=3204, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1324KiB/s (1356kB/s), 329KiB/s-336KiB/s (337kB/s-344kB/s), io=19.6MiB (20.5MB), run=15053-15159msec
  WRITE: bw=1378KiB/s (1411kB/s), 344KiB/s-348KiB/s (352kB/s-356kB/s), io=20.4MiB (21.4MB), run=15053-15159msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=21923: Wed Apr 17 18:28:26 2024
   read: IOPS=10, BW=435MiB/s (456MB/s)(40.0MiB/92msec)
    slat (nsec): min=7447.0k, max=7447.0k, avg=7447960.00, stdev= 0.00
    clat (nsec): min=80755k, max=80755k, avg=80755157.00, stdev= 0.00
     lat (nsec): min=88212k, max=88212k, avg=88212202.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[81265],  5.00th=[81265], 10.00th=[81265], 20.00th=[81265],
     | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265],
     | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265],
     | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.70%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=435MiB/s (456MB/s), 435MiB/s-435MiB/s (456MB/s-456MB/s), io=40.0MiB (41.9MB), run=92-92msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck