== sanity test 398c: run fio to test AIO ================= 22:35:25 (1713494125)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0967855 s, 433 MB/s
osc.lustre-OST0000-osc-ffff88012c001800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=32581: Thu Apr 18 22:36:01 2024
  write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34340msec)
    slat (usec): min=12, max=596, avg=84.00, stdev=55.82
    clat (msec): min=65, max=343, avg=214.17, stdev=40.74
     lat (msec): min=65, max=343, avg=214.26, stdev=40.75
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  148], 10.00th=[  163], 20.00th=[  184],
     | 30.00th=[  197], 40.00th=[  207], 50.00th=[  215], 60.00th=[  226],
     | 70.00th=[  236], 80.00th=[  249], 90.00th=[  264], 95.00th=[  275],
     | 99.00th=[  300], 99.50th=[  313], 99.90th=[  321], 99.95th=[  330],
     | 99.99th=[  342]
   bw (  KiB/s): min=  208, max=  512, per=24.91%, avg=296.94, stdev=48.02, samples=68
   iops        : min=   52, max=  128, avg=74.15, stdev=12.02, samples=68
  lat (msec)   : 100=0.74%, 250=79.84%, 500=19.41%
  cpu          : usr=0.13%, sys=0.77%, ctx=1953, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32582: Thu Apr 18 22:36:01 2024
  write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34340msec)
    slat (usec): min=13, max=564, avg=82.44, stdev=53.80
    clat (msec): min=65, max=344, avg=214.19, stdev=40.24
     lat (msec): min=65, max=344, avg=214.27, stdev=40.25
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  148], 10.00th=[  165], 20.00th=[  184],
     | 30.00th=[  197], 40.00th=[  205], 50.00th=[  215], 60.00th=[  224],
     | 70.00th=[  236], 80.00th=[  249], 90.00th=[  264], 95.00th=[  279],
     | 99.00th=[  300], 99.50th=[  313], 99.90th=[  330], 99.95th=[  330],
     | 99.99th=[  347]
   bw (  KiB/s): min=  216, max=  512, per=24.90%, avg=296.82, stdev=48.12, samples=68
   iops        : min=   54, max=  128, avg=74.12, stdev=12.05, samples=68
  lat (msec)   : 100=0.74%, 250=80.39%, 500=18.87%
  cpu          : usr=0.16%, sys=0.71%, ctx=2084, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32583: Thu Apr 18 22:36:01 2024
  write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34329msec)
    slat (usec): min=13, max=592, avg=84.37, stdev=56.20
    clat (msec): min=65, max=343, avg=214.10, stdev=40.16
     lat (msec): min=65, max=343, avg=214.18, stdev=40.17
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  148], 10.00th=[  163], 20.00th=[  184],
     | 30.00th=[  197], 40.00th=[  205], 50.00th=[  215], 60.00th=[  226],
     | 70.00th=[  234], 80.00th=[  249], 90.00th=[  264], 95.00th=[  275],
     | 99.00th=[  300], 99.50th=[  309], 99.90th=[  334], 99.95th=[  334],
     | 99.99th=[  342]
   bw (  KiB/s): min=  224, max=  512, per=24.94%, avg=297.29, stdev=47.07, samples=68
   iops        : min=   56, max=  128, avg=74.24, stdev=11.80, samples=68
  lat (msec)   : 100=0.74%, 250=81.02%, 500=18.24%
  cpu          : usr=0.11%, sys=0.78%, ctx=2098, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32584: Thu Apr 18 22:36:01 2024
  write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34360msec)
    slat (usec): min=12, max=596, avg=82.86, stdev=53.59
    clat (msec): min=30, max=343, avg=214.33, stdev=40.35
     lat (msec): min=30, max=343, avg=214.41, stdev=40.36
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  150], 10.00th=[  163], 20.00th=[  182],
     | 30.00th=[  194], 40.00th=[  207], 50.00th=[  215], 60.00th=[  226],
     | 70.00th=[  234], 80.00th=[  249], 90.00th=[  266], 95.00th=[  275],
     | 99.00th=[  309], 99.50th=[  317], 99.90th=[  334], 99.95th=[  334],
     | 99.99th=[  342]
   bw (  KiB/s): min=  184, max=  512, per=24.86%, avg=296.28, stdev=49.46, samples=68
   iops        : min=   46, max=  128, avg=74.01, stdev=12.37, samples=68
  lat (msec)   : 50=0.08%, 100=0.31%, 250=80.04%, 500=19.57%
  cpu          : usr=0.17%, sys=0.72%, ctx=2030, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1192KiB/s (1221kB/s), 298KiB/s-298KiB/s (305kB/s-305kB/s), io=40.0MiB (41.9MB), run=34329-34360msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=32621: Thu Apr 18 22:36:22 2024
   read: IOPS=62, BW=251KiB/s (257kB/s)(5048KiB/20090msec)
    slat (usec): min=729, max=4551, avg=1550.44, stdev=390.45
    clat (msec): min=3, max=132, avg=81.55, stdev=16.57
     lat (msec): min=4, max=133, avg=83.11, stdev=16.65
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   58], 10.00th=[   64], 20.00th=[   69],
     | 30.00th=[   73], 40.00th=[   78], 50.00th=[   81], 60.00th=[   86],
     | 70.00th=[   89], 80.00th=[   95], 90.00th=[  104], 95.00th=[  110],
     | 99.00th=[  122], 99.50th=[  127], 99.90th=[  132], 99.95th=[  133],
     | 99.99th=[  133]
   bw (  KiB/s): min=  136, max=  352, per=25.06%, avg=250.05, stdev=53.27, samples=40
   iops        : min=   34, max=   88, avg=62.40, stdev=13.29, samples=40
  write: IOPS=64, BW=258KiB/s (265kB/s)(5192KiB/20090msec)
    slat (usec): min=14, max=2005, avg=89.25, stdev=86.28
    clat (msec): min=68, max=322, avg=166.31, stdev=39.22
     lat (msec): min=68, max=322, avg=166.40, stdev=39.23
    clat percentiles (msec):
     |  1.00th=[   93],  5.00th=[  108], 10.00th=[  118], 20.00th=[  134],
     | 30.00th=[  144], 40.00th=[  155], 50.00th=[  163], 60.00th=[  174],
     | 70.00th=[  184], 80.00th=[  197], 90.00th=[  215], 95.00th=[  241],
     | 99.00th=[  279], 99.50th=[  292], 99.90th=[  321], 99.95th=[  321],
     | 99.99th=[  321]
   bw (  KiB/s): min=  191, max=  344, per=24.80%, avg=257.65, stdev=38.69, samples=40
   iops        : min=   47, max=   86, avg=64.30, stdev= 9.67, samples=40
  lat (msec)   : 4=0.04%, 20=0.39%, 50=0.35%, 100=42.85%, 250=54.49%
  lat (msec)   : 500=1.88%
  cpu          : usr=0.30%, sys=2.05%, ctx=3496, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32622: Thu Apr 18 22:36:22 2024
   read: IOPS=61, BW=248KiB/s (254kB/s)(4976KiB/20090msec)
    slat (usec): min=778, max=3790, avg=1550.70, stdev=365.37
    clat (msec): min=2, max=132, avg=81.69, stdev=16.53
     lat (msec): min=3, max=134, avg=83.24, stdev=16.60
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   59], 10.00th=[   64], 20.00th=[   70],
     | 30.00th=[   73], 40.00th=[   78], 50.00th=[   82], 60.00th=[   85],
     | 70.00th=[   90], 80.00th=[   95], 90.00th=[  103], 95.00th=[  109],
     | 99.00th=[  128], 99.50th=[  130], 99.90th=[  132], 99.95th=[  132],
     | 99.99th=[  132]
   bw (  KiB/s): min=   96, max=  352, per=24.69%, avg=246.43, stdev=54.10, samples=40
   iops        : min=   24, max=   88, avg=61.48, stdev=13.50, samples=40
  write: IOPS=65, BW=262KiB/s (268kB/s)(5264KiB/20090msec)
    slat (usec): min=13, max=580, avg=87.33, stdev=47.88
    clat (msec): min=44, max=324, avg=165.19, stdev=40.22
     lat (msec): min=44, max=325, avg=165.28, stdev=40.22
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  107], 10.00th=[  117], 20.00th=[  131],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  163], 60.00th=[  174],
     | 70.00th=[  182], 80.00th=[  197], 90.00th=[  218], 95.00th=[  243],
     | 99.00th=[  275], 99.50th=[  288], 99.90th=[  321], 99.95th=[  326],
     | 99.99th=[  326]
   bw (  KiB/s): min=  168, max=  359, per=25.16%, avg=261.43, stdev=42.37, samples=40
   iops        : min=   42, max=   89, avg=65.22, stdev=10.59, samples=40
  lat (msec)   : 4=0.12%, 20=0.31%, 50=0.43%, 100=43.40%, 250=53.67%
  lat (msec)   : 500=2.07%
  cpu          : usr=0.29%, sys=2.03%, ctx=3485, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32623: Thu Apr 18 22:36:22 2024
   read: IOPS=62, BW=248KiB/s (254kB/s)(4984KiB/20090msec)
    slat (usec): min=746, max=3637, avg=1531.58, stdev=352.06
    clat (usec): min=1926, max=132838, avg=82204.54, stdev=16469.91
     lat (msec): min=3, max=134, avg=83.74, stdev=16.53
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   58], 10.00th=[   63], 20.00th=[   70],
     | 30.00th=[   74], 40.00th=[   79], 50.00th=[   83], 60.00th=[   86],
     | 70.00th=[   90], 80.00th=[   95], 90.00th=[  103], 95.00th=[  110],
     | 99.00th=[  126], 99.50th=[  130], 99.90th=[  133], 99.95th=[  133],
     | 99.99th=[  133]
   bw (  KiB/s): min=  160, max=  344, per=24.81%, avg=247.65, stdev=45.52, samples=40
   iops        : min=   40, max=   86, avg=61.80, stdev=11.38, samples=40
  write: IOPS=65, BW=262KiB/s (268kB/s)(5256KiB/20090msec)
    slat (usec): min=15, max=575, avg=88.19, stdev=50.83
    clat (msec): min=68, max=325, avg=164.74, stdev=39.01
     lat (msec): min=68, max=325, avg=164.83, stdev=39.00
    clat percentiles (msec):
     |  1.00th=[   93],  5.00th=[  108], 10.00th=[  117], 20.00th=[  133],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  161], 60.00th=[  171],
     | 70.00th=[  182], 80.00th=[  194], 90.00th=[  215], 95.00th=[  232],
     | 99.00th=[  275], 99.50th=[  292], 99.90th=[  321], 99.95th=[  326],
     | 99.99th=[  326]
   bw (  KiB/s): min=  144, max=  336, per=25.01%, avg=259.85, stdev=45.31, samples=40
   iops        : min=   36, max=   84, avg=64.85, stdev=11.34, samples=40
  lat (msec)   : 2=0.04%, 20=0.27%, 50=0.39%, 100=42.93%, 250=54.69%
  lat (msec)   : 500=1.68%
  cpu          : usr=0.27%, sys=2.07%, ctx=3372, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32624: Thu Apr 18 22:36:22 2024
   read: IOPS=63, BW=253KiB/s (259kB/s)(5060KiB/19988msec)
    slat (usec): min=30, max=3710, avg=1524.48, stdev=372.05
    clat (usec): min=1943, max=161942, avg=81724.03, stdev=16367.76
     lat (msec): min=2, max=163, avg=83.25, stdev=16.47
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   59], 10.00th=[   64], 20.00th=[   70],
     | 30.00th=[   73], 40.00th=[   78], 50.00th=[   82], 60.00th=[   85],
     | 70.00th=[   90], 80.00th=[   94], 90.00th=[  104], 95.00th=[  109],
     | 99.00th=[  124], 99.50th=[  128], 99.90th=[  134], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  112, max=  408, per=25.41%, avg=253.62, stdev=57.64, samples=39
   iops        : min=   28, max=  102, avg=63.31, stdev=14.46, samples=39
  write: IOPS=64, BW=259KiB/s (265kB/s)(5180KiB/19988msec)
    slat (usec): min=13, max=433, avg=85.40, stdev=45.98
    clat (msec): min=33, max=321, avg=165.38, stdev=41.16
     lat (msec): min=34, max=321, avg=165.46, stdev=41.15
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[  106], 10.00th=[  117], 20.00th=[  133],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  163], 60.00th=[  171],
     | 70.00th=[  182], 80.00th=[  194], 90.00th=[  218], 95.00th=[  245],
     | 99.00th=[  288], 99.50th=[  296], 99.90th=[  309], 99.95th=[  321],
     | 99.99th=[  321]
   bw (  KiB/s): min=  160, max=  344, per=24.78%, avg=257.51, stdev=42.07, samples=39
   iops        : min=   40, max=   86, avg=64.28, stdev=10.50, samples=39
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.12%, 20=0.16%, 50=0.35%
  lat (msec)   : 100=44.14%, 250=52.93%, 500=2.23%
  cpu          : usr=0.29%, sys=2.07%, ctx=3320, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=999KiB/s (1023kB/s), 248KiB/s-253KiB/s (254kB/s-259kB/s), io=19.6MiB (20.5MB), run=19988-20090msec
  WRITE: bw=1040KiB/s (1065kB/s), 258KiB/s-262KiB/s (265kB/s-268kB/s), io=20.4MiB (21.4MB), run=19988-20090msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=32644: Thu Apr 18 22:36:23 2024
   read: IOPS=5, BW=208MiB/s (218MB/s)(40.0MiB/192msec)
    slat (nsec): min=5976.2k, max=5976.2k, avg=5976230.00, stdev= 0.00
    clat (nsec): min=179606k, max=179606k, avg=179606447.00, stdev= 0.00
     lat (nsec): min=185592k, max=185592k, avg=185592222.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  180],  5.00th=[  180], 10.00th=[  180], 20.00th=[  180],
     | 30.00th=[  180], 40.00th=[  180], 50.00th=[  180], 60.00th=[  180],
     | 70.00th=[  180], 80.00th=[  180], 90.00th=[  180], 95.00th=[  180],
     | 99.00th=[  180], 99.50th=[  180], 99.90th=[  180], 99.95th=[  180],
     | 99.99th=[  180]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=2.62%, ctx=17, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=40.0MiB (41.9MB), run=192-192msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout