== sanity test 398c: run fio to test AIO ================= 17:14:37 (1713302077)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.121757 s, 344 MB/s
osc.lustre-OST0000-osc-ffff8800ac28b000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=32442: Tue Apr 16 17:15:09 2024
  write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31017msec)
    slat (usec): min=12, max=350, avg=54.41, stdev=33.65
    clat (msec): min=45, max=294, avg=193.37, stdev=32.93
     lat (msec): min=45, max=294, avg=193.43, stdev=32.93
    clat percentiles (msec):
     |  1.00th=[  115],  5.00th=[  144], 10.00th=[  155], 20.00th=[  169],
     | 30.00th=[  178], 40.00th=[  184], 50.00th=[  192], 60.00th=[  201],
     | 70.00th=[  211], 80.00th=[  222], 90.00th=[  234], 95.00th=[  247],
     | 99.00th=[  268], 99.50th=[  275], 99.90th=[  292], 99.95th=[  292],
     | 99.99th=[  296]
   bw (  KiB/s): min=  255, max=  400, per=24.87%, avg=328.26, stdev=38.61, samples=62
   iops        : min=   63, max=  100, avg=81.97, stdev= 9.70, samples=62
  lat (msec)   : 50=0.39%, 100=0.47%, 250=95.08%, 500=4.06%
  cpu          : usr=0.11%, sys=0.57%, ctx=2261, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32443: Tue Apr 16 17:15:09 2024
  write: IOPS=82, BW=331KiB/s (338kB/s)(10.0MiB/30982msec)
    slat (usec): min=13, max=440, avg=54.55, stdev=34.43
    clat (msec): min=44, max=351, avg=193.12, stdev=33.42
     lat (msec): min=45, max=351, avg=193.18, stdev=33.42
    clat percentiles (msec):
     |  1.00th=[  120],  5.00th=[  144], 10.00th=[  157], 20.00th=[  169],
     | 30.00th=[  178], 40.00th=[  184], 50.00th=[  192], 60.00th=[  199],
     | 70.00th=[  207], 80.00th=[  220], 90.00th=[  236], 95.00th=[  249],
     | 99.00th=[  275], 99.50th=[  284], 99.90th=[  313], 99.95th=[  330],
     | 99.99th=[  351]
   bw (  KiB/s): min=  255, max=  424, per=25.05%, avg=330.62, stdev=39.04, samples=61
   iops        : min=   63, max=  106, avg=82.56, stdev= 9.78, samples=61
  lat (msec)   : 50=0.62%, 100=0.08%, 250=94.84%, 500=4.45%
  cpu          : usr=0.08%, sys=0.59%, ctx=2192, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32444: Tue Apr 16 17:15:09 2024
  write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/30995msec)
    slat (usec): min=12, max=337, avg=54.62, stdev=33.31
    clat (msec): min=44, max=328, avg=193.21, stdev=33.23
     lat (msec): min=45, max=328, avg=193.26, stdev=33.23
    clat percentiles (msec):
     |  1.00th=[  113],  5.00th=[  142], 10.00th=[  155], 20.00th=[  169],
     | 30.00th=[  178], 40.00th=[  184], 50.00th=[  192], 60.00th=[  201],
     | 70.00th=[  209], 80.00th=[  222], 90.00th=[  234], 95.00th=[  247],
     | 99.00th=[  271], 99.50th=[  275], 99.90th=[  313], 99.95th=[  317],
     | 99.99th=[  330]
   bw (  KiB/s): min=  255, max=  424, per=25.03%, avg=330.36, stdev=34.88, samples=61
   iops        : min=   63, max=  106, avg=82.49, stdev= 8.76, samples=61
  lat (msec)   : 50=0.62%, 100=0.23%, 250=95.04%, 500=4.10%
  cpu          : usr=0.08%, sys=0.59%, ctx=2343, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32445: Tue Apr 16 17:15:09 2024
  write: IOPS=82, BW=331KiB/s (338kB/s)(10.0MiB/30982msec)
    slat (usec): min=13, max=310, avg=55.38, stdev=33.56
    clat (msec): min=45, max=404, avg=193.11, stdev=33.49
     lat (msec): min=45, max=404, avg=193.17, stdev=33.49
    clat percentiles (msec):
     |  1.00th=[  123],  5.00th=[  146], 10.00th=[  157], 20.00th=[  167],
     | 30.00th=[  178], 40.00th=[  184], 50.00th=[  192], 60.00th=[  199],
     | 70.00th=[  209], 80.00th=[  220], 90.00th=[  234], 95.00th=[  247],
     | 99.00th=[  275], 99.50th=[  296], 99.90th=[  380], 99.95th=[  393],
     | 99.99th=[  405]
   bw (  KiB/s): min=  256, max=  400, per=25.07%, avg=330.89, stdev=35.88, samples=61
   iops        : min=   64, max=  100, avg=82.62, stdev= 9.00, samples=61
  lat (msec)   : 50=0.62%, 100=0.08%, 250=95.08%, 500=4.22%
  cpu          : usr=0.09%, sys=0.58%, ctx=2289, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1321KiB/s (1352kB/s), 330KiB/s-331KiB/s (338kB/s-338kB/s), io=40.0MiB (41.9MB), run=30982-31017msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=32480: Tue Apr 16 17:15:29 2024
   read: IOPS=66, BW=265KiB/s (271kB/s)(5048KiB/19042msec)
    slat (usec): min=729, max=3193, avg=1263.02, stdev=308.80
    clat (msec): min=2, max=197, avg=77.89, stdev=14.80
     lat (msec): min=3, max=198, avg=79.16, stdev=14.85
    clat percentiles (msec):
     |  1.00th=[   50],  5.00th=[   56], 10.00th=[   63], 20.00th=[   67],
     | 30.00th=[   71], 40.00th=[   75], 50.00th=[   79], 60.00th=[   81],
     | 70.00th=[   84], 80.00th=[   88], 90.00th=[   94], 95.00th=[  101],
     | 99.00th=[  129], 99.50th=[  138], 99.90th=[  155], 99.95th=[  199],
     | 99.99th=[  199]
   bw (  KiB/s): min=  176, max=  344, per=25.20%, avg=264.32, stdev=41.38, samples=38
   iops        : min=   44, max=   86, avg=66.00, stdev=10.31, samples=38
  write: IOPS=68, BW=273KiB/s (279kB/s)(5192KiB/19042msec)
    slat (usec): min=12, max=1289, avg=49.73, stdev=43.01
    clat (msec): min=68, max=322, avg=157.29, stdev=33.28
     lat (msec): min=68, max=323, avg=157.34, stdev=33.28
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[  107], 10.00th=[  113], 20.00th=[  129],
     | 30.00th=[  140], 40.00th=[  148], 50.00th=[  157], 60.00th=[  165],
     | 70.00th=[  176], 80.00th=[  184], 90.00th=[  199], 95.00th=[  211],
     | 99.00th=[  239], 99.50th=[  251], 99.90th=[  309], 99.95th=[  326],
     | 99.99th=[  326]
   bw (  KiB/s): min=  160, max=  328, per=24.84%, avg=271.26, stdev=32.28, samples=38
   iops        : min=   40, max=   82, avg=67.74, stdev= 8.06, samples=38
  lat (msec)   : 4=0.04%, 20=0.16%, 50=0.59%, 100=47.81%, 250=51.17%
  lat (msec)   : 500=0.23%
  cpu          : usr=0.11%, sys=1.28%, ctx=3263, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32481: Tue Apr 16 17:15:29 2024
   read: IOPS=65, BW=260KiB/s (267kB/s)(4976KiB/19119msec)
    slat (usec): min=712, max=2982, avg=1266.77, stdev=289.36
    clat (msec): min=2, max=149, avg=77.88, stdev=14.09
     lat (msec): min=3, max=152, avg=79.15, stdev=14.15
    clat percentiles (msec):
     |  1.00th=[   51],  5.00th=[   59], 10.00th=[   64], 20.00th=[   67],
     | 30.00th=[   71], 40.00th=[   75], 50.00th=[   79], 60.00th=[   81],
     | 70.00th=[   84], 80.00th=[   88], 90.00th=[   94], 95.00th=[  100],
     | 99.00th=[  113], 99.50th=[  132], 99.90th=[  148], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=  168, max=  352, per=24.74%, avg=259.47, stdev=48.49, samples=38
   iops        : min=   42, max=   88, avg=64.79, stdev=12.09, samples=38
  write: IOPS=68, BW=275KiB/s (282kB/s)(5264KiB/19119msec)
    slat (usec): min=13, max=1648, avg=50.26, stdev=50.17
    clat (msec): min=61, max=272, avg=157.28, stdev=33.28
     lat (msec): min=61, max=272, avg=157.33, stdev=33.28
    clat percentiles (msec):
     |  1.00th=[   88],  5.00th=[  104], 10.00th=[  116], 20.00th=[  130],
     | 30.00th=[  140], 40.00th=[  148], 50.00th=[  157], 60.00th=[  163],
     | 70.00th=[  174], 80.00th=[  182], 90.00th=[  201], 95.00th=[  218],
     | 99.00th=[  239], 99.50th=[  253], 99.90th=[  264], 99.95th=[  275],
     | 99.99th=[  275]
   bw (  KiB/s): min=  200, max=  352, per=25.09%, avg=274.00, stdev=39.32, samples=38
   iops        : min=   50, max=   88, avg=68.42, stdev= 9.84, samples=38
  lat (msec)   : 4=0.04%, 20=0.23%, 50=0.20%, 100=48.01%, 250=51.17%
  lat (msec)   : 500=0.35%
  cpu          : usr=0.12%, sys=1.26%, ctx=3460, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32482: Tue Apr 16 17:15:29 2024
   read: IOPS=65, BW=261KiB/s (268kB/s)(4984KiB/19069msec)
    slat (usec): min=30, max=3600, avg=1246.41, stdev=315.02
    clat (usec): min=1729, max=223269, avg=77352.33, stdev=15152.25
     lat (msec): min=2, max=224, avg=78.60, stdev=15.20
    clat percentiles (msec):
     |  1.00th=[   25],  5.00th=[   56], 10.00th=[   63], 20.00th=[   67],
     | 30.00th=[   70], 40.00th=[   75], 50.00th=[   79], 60.00th=[   82],
     | 70.00th=[   84], 80.00th=[   89], 90.00th=[   94], 95.00th=[  100],
     | 99.00th=[  110], 99.50th=[  116], 99.90th=[  142], 99.95th=[  224],
     | 99.99th=[  224]
   bw (  KiB/s): min=  168, max=  368, per=24.96%, avg=261.82, stdev=51.02, samples=38
   iops        : min=   42, max=   92, avg=65.39, stdev=12.72, samples=38
  write: IOPS=68, BW=276KiB/s (282kB/s)(5256KiB/19069msec)
    slat (usec): min=13, max=241, avg=49.12, stdev=24.87
    clat (msec): min=35, max=309, avg=157.41, stdev=34.76
     lat (msec): min=35, max=309, avg=157.46, stdev=34.76
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[  106], 10.00th=[  116], 20.00th=[  129],
     | 30.00th=[  140], 40.00th=[  148], 50.00th=[  157], 60.00th=[  165],
     | 70.00th=[  174], 80.00th=[  184], 90.00th=[  203], 95.00th=[  220],
     | 99.00th=[  251], 99.50th=[  253], 99.90th=[  279], 99.95th=[  309],
     | 99.99th=[  309]
   bw (  KiB/s): min=  192, max=  384, per=25.07%, avg=273.82, stdev=36.43, samples=38
   iops        : min=   48, max=   96, avg=68.39, stdev= 9.12, samples=38
  lat (msec)   : 2=0.04%, 4=0.16%, 10=0.08%, 20=0.20%, 50=0.66%
  lat (msec)   : 100=47.50%, 250=50.82%, 500=0.55%
  cpu          : usr=0.13%, sys=1.27%, ctx=3423, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32483: Tue Apr 16 17:15:29 2024
   read: IOPS=66, BW=265KiB/s (271kB/s)(5060KiB/19092msec)
    slat (usec): min=699, max=3143, avg=1244.91, stdev=297.01
    clat (usec): min=1786, max=160135, avg=77367.97, stdev=15143.86
     lat (msec): min=3, max=161, avg=78.61, stdev=15.21
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   56], 10.00th=[   63], 20.00th=[   66],
     | 30.00th=[   69], 40.00th=[   74], 50.00th=[   78], 60.00th=[   81],
     | 70.00th=[   84], 80.00th=[   89], 90.00th=[   95], 95.00th=[  101],
     | 99.00th=[  125], 99.50th=[  140], 99.90th=[  148], 99.95th=[  161],
     | 99.99th=[  161]
   bw (  KiB/s): min=  160, max=  392, per=25.32%, avg=265.63, stdev=54.33, samples=38
   iops        : min=   40, max=   98, avg=66.37, stdev=13.54, samples=38
  write: IOPS=67, BW=271KiB/s (278kB/s)(5180KiB/19092msec)
    slat (usec): min=13, max=440, avg=49.42, stdev=28.51
    clat (msec): min=74, max=333, avg=158.67, stdev=35.14
     lat (msec): min=74, max=333, avg=158.72, stdev=35.14
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[  106], 10.00th=[  116], 20.00th=[  131],
     | 30.00th=[  140], 40.00th=[  148], 50.00th=[  157], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  205], 95.00th=[  218],
     | 99.00th=[  249], 99.50th=[  268], 99.90th=[  334], 99.95th=[  334],
     | 99.99th=[  334]
   bw (  KiB/s): min=  200, max=  344, per=24.73%, avg=270.05, stdev=32.34, samples=38
   iops        : min=   50, max=   86, avg=67.47, stdev= 8.09, samples=38
  lat (msec)   : 2=0.04%, 20=0.31%, 50=0.62%, 100=47.50%, 250=51.05%
  lat (msec)   : 500=0.47%
  cpu          : usr=0.13%, sys=1.23%, ctx=3500, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1050KiB/s (1075kB/s), 260KiB/s-265KiB/s (267kB/s-271kB/s), io=19.6MiB (20.5MB), run=19042-19119msec
  WRITE: bw=1093KiB/s (1119kB/s), 271KiB/s-276KiB/s (278kB/s-282kB/s), io=20.4MiB (21.4MB), run=19042-19119msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=32502: Tue Apr 16 17:15:31 2024
   read: IOPS=5, BW=221MiB/s (232MB/s)(40.0MiB/181msec)
    slat (nsec): min=7943.4k, max=7943.4k, avg=7943445.00, stdev= 0.00
    clat (nsec): min=166284k, max=166284k, avg=166283716.00, stdev= 0.00
     lat (nsec): min=174236k, max=174236k, avg=174236161.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  167],  5.00th=[  167], 10.00th=[  167], 20.00th=[  167],
     | 30.00th=[  167], 40.00th=[  167], 50.00th=[  167], 60.00th=[  167],
     | 70.00th=[  167], 80.00th=[  167], 90.00th=[  167], 95.00th=[  167],
     | 99.00th=[  167], 99.50th=[  167], 99.90th=[  167], 99.95th=[  167],
     | 99.99th=[  167]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.89%, ctx=13, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=40.0MiB (41.9MB), run=181-181msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout