== sanity test 398c: run fio to test AIO ================= 03:39:40 (1713512380)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0751745 s, 558 MB/s
osc.lustre-OST0000-osc-ffff880097e2a800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16123: Fri Apr 19 03:39:44 2024
  write: IOPS=981, BW=3925KiB/s (4019kB/s)(10.0MiB/2609msec)
    slat (usec): min=11, max=300, avg=41.76, stdev=23.76
    clat (usec): min=2446, max=31910, avg=16218.26, stdev=2136.30
     lat (usec): min=2487, max=31958, avg=16261.14, stdev=2137.53
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12911], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16188], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18220], 95.00th=[19268],
     | 99.00th=[25035], 99.50th=[26346], 99.90th=[29754], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 3728, max= 4079, per=25.13%, avg=3945.40, stdev=130.15, samples=5
   iops        : min=  932, max= 1019, avg=986.20, stdev=32.35, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=96.64%, 50=3.16%
  cpu          : usr=0.73%, sys=4.64%, ctx=1602, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16124: Fri Apr 19 03:39:44 2024
  write: IOPS=981, BW=3926KiB/s (4021kB/s)(10.0MiB/2608msec)
    slat (usec): min=11, max=19742, avg=46.80, stdev=390.04
    clat (usec): min=3605, max=30415, avg=16187.83, stdev=2077.09
     lat (usec): min=3744, max=30478, avg=16235.72, stdev=2068.67
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12780], 10.00th=[13960], 20.00th=[14877],
     | 30.00th=[15533], 40.00th=[15795], 50.00th=[16188], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18220], 95.00th=[19268],
     | 99.00th=[23200], 99.50th=[26608], 99.90th=[30278], 99.95th=[30278],
     | 99.99th=[30540]
   bw (  KiB/s): min= 3712, max= 4095, per=25.14%, avg=3947.00, stdev=141.45, samples=5
   iops        : min=  928, max= 1023, avg=986.60, stdev=35.17, samples=5
  lat (msec)   : 4=0.08%, 10=0.20%, 20=96.95%, 50=2.77%
  cpu          : usr=0.77%, sys=4.99%, ctx=1384, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16125: Fri Apr 19 03:39:44 2024
  write: IOPS=984, BW=3937KiB/s (4031kB/s)(10.0MiB/2601msec)
    slat (usec): min=11, max=293, avg=43.90, stdev=24.86
    clat (usec): min=2909, max=43321, avg=16180.27, stdev=2357.46
     lat (usec): min=2957, max=43342, avg=16225.25, stdev=2358.13
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12911], 10.00th=[13960], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16057], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18220], 95.00th=[19268],
     | 99.00th=[23200], 99.50th=[28705], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[43254]
   bw (  KiB/s): min= 3688, max= 4096, per=25.23%, avg=3961.40, stdev=163.71, samples=5
   iops        : min=  922, max= 1024, avg=990.20, stdev=40.79, samples=5
  lat (msec)   : 4=0.08%, 10=0.62%, 20=96.21%, 50=3.09%
  cpu          : usr=0.62%, sys=5.04%, ctx=1858, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16126: Fri Apr 19 03:39:44 2024
  write: IOPS=981, BW=3926KiB/s (4021kB/s)(10.0MiB/2608msec)
    slat (usec): min=12, max=195, avg=39.66, stdev=21.76
    clat (usec): min=8184, max=32813, avg=16210.69, stdev=2081.62
     lat (usec): min=8261, max=32860, avg=16251.44, stdev=2083.33
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12780], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16057], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18220], 95.00th=[19530],
     | 99.00th=[22938], 99.50th=[26608], 99.90th=[30278], 99.95th=[30540],
     | 99.99th=[32900]
   bw (  KiB/s): min= 3728, max= 4071, per=25.13%, avg=3945.40, stdev=129.17, samples=5
   iops        : min=  932, max= 1017, avg=986.20, stdev=32.11, samples=5
  lat (msec)   : 10=0.43%, 20=96.09%, 50=3.48%
  cpu          : usr=0.81%, sys=4.30%, ctx=1422, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=15.3MiB/s (16.1MB/s), 3925KiB/s-3937KiB/s (4019kB/s-4031kB/s), io=40.0MiB (41.9MB), run=2601-2609msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16138: Fri Apr 19 03:39:46 2024
   read: IOPS=622, BW=2489KiB/s (2549kB/s)(5048KiB/2028msec)
    slat (usec): min=441, max=2315, avg=769.48, stdev=224.19
    clat (usec): min=2353, max=33027, avg=11964.58, stdev=3037.39
     lat (usec): min=3179, max=34807, avg=12735.82, stdev=3191.35
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[ 8848], 10.00th=[ 9241], 20.00th=[10028],
     | 30.00th=[10552], 40.00th=[11076], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12387], 80.00th=[13042], 90.00th=[14222], 95.00th=[16909],
     | 99.00th=[25297], 99.50th=[27657], 99.90th=[31589], 99.95th=[32900],
     | 99.99th=[32900]
   bw (  KiB/s): min= 1876, max= 2840, per=24.97%, avg=2471.00, stdev=417.90, samples=4
   iops        : min=  469, max=  710, avg=617.75, stdev=104.47, samples=4
  write: IOPS=640, BW=2560KiB/s (2622kB/s)(5192KiB/2028msec)
    slat (usec): min=12, max=957, avg=41.93, stdev=41.62
    clat (usec): min=6825, max=31626, avg=12360.05, stdev=3192.39
     lat (usec): min=6870, max=31670, avg=12403.05, stdev=3200.56
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9634], 10.00th=[10028], 20.00th=[10683],
     | 30.00th=[11076], 40.00th=[11469], 50.00th=[11863], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13829], 95.00th=[18482],
     | 99.00th=[28967], 99.50th=[30540], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 2083, max= 2800, per=24.72%, avg=2546.75, stdev=317.76, samples=4
   iops        : min=  520, max=  700, avg=636.50, stdev=79.81, samples=4
  lat (msec)   : 4=0.04%, 10=14.49%, 20=81.60%, 50=3.87%
  cpu          : usr=1.28%, sys=8.83%, ctx=2726, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16139: Fri Apr 19 03:39:46 2024
   read: IOPS=616, BW=2467KiB/s (2526kB/s)(4976KiB/2017msec)
    slat (usec): min=34, max=2194, avg=766.95, stdev=229.06
    clat (usec): min=1363, max=29964, avg=11952.09, stdev=3135.55
     lat (usec): min=1477, max=31559, avg=12720.90, stdev=3298.50
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8848], 10.00th=[ 9372], 20.00th=[10028],
     | 30.00th=[10552], 40.00th=[10945], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12387], 80.00th=[13042], 90.00th=[14091], 95.00th=[18220],
     | 99.00th=[25560], 99.50th=[27919], 99.90th=[29492], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1964, max= 2848, per=24.83%, avg=2457.00, stdev=366.03, samples=4
   iops        : min=  491, max=  712, avg=614.25, stdev=91.51, samples=4
  write: IOPS=652, BW=2610KiB/s (2672kB/s)(5264KiB/2017msec)
    slat (usec): min=12, max=789, avg=42.19, stdev=36.28
    clat (usec): min=3548, max=30878, avg=12378.55, stdev=3008.17
     lat (usec): min=3627, max=30967, avg=12421.91, stdev=3018.15
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[ 9765], 10.00th=[10290], 20.00th=[10683],
     | 30.00th=[11076], 40.00th=[11469], 50.00th=[11863], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13829], 95.00th=[18744],
     | 99.00th=[27919], 99.50th=[29492], 99.90th=[30802], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 2179, max= 2816, per=25.34%, avg=2610.75, stdev=291.98, samples=4
   iops        : min=  544, max=  704, avg=652.50, stdev=73.36, samples=4
  lat (msec)   : 2=0.08%, 4=0.12%, 10=13.52%, 20=82.19%, 50=4.10%
  cpu          : usr=1.09%, sys=9.38%, ctx=2608, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16140: Fri Apr 19 03:39:46 2024
   read: IOPS=615, BW=2462KiB/s (2522kB/s)(4984KiB/2024msec)
    slat (usec): min=464, max=2159, avg=767.90, stdev=221.37
    clat (usec): min=2839, max=29923, avg=11913.79, stdev=2971.01
     lat (usec): min=3459, max=31459, avg=12683.34, stdev=3129.65
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 8717], 10.00th=[ 9372], 20.00th=[10028],
     | 30.00th=[10552], 40.00th=[11076], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12387], 80.00th=[12911], 90.00th=[14091], 95.00th=[16909],
     | 99.00th=[25035], 99.50th=[25822], 99.90th=[28705], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 2027, max= 2688, per=24.83%, avg=2456.75, stdev=301.10, samples=4
   iops        : min=  506, max=  672, avg=614.00, stdev=75.63, samples=4
  write: IOPS=649, BW=2597KiB/s (2659kB/s)(5256KiB/2024msec)
    slat (usec): min=12, max=741, avg=43.02, stdev=31.49
    clat (usec): min=7204, max=32038, avg=12390.54, stdev=3218.35
     lat (usec): min=7246, max=32066, avg=12434.61, stdev=3227.84
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9765], 10.00th=[10159], 20.00th=[10683],
     | 30.00th=[11076], 40.00th=[11469], 50.00th=[11731], 60.00th=[12125],
     | 70.00th=[12387], 80.00th=[12911], 90.00th=[13829], 95.00th=[19006],
     | 99.00th=[27919], 99.50th=[30540], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1964, max= 2976, per=25.09%, avg=2585.00, stdev=442.83, samples=4
   iops        : min=  491, max=  744, avg=646.25, stdev=110.71, samples=4
  lat (msec)   : 4=0.08%, 10=13.63%, 20=82.07%, 50=4.22%
  cpu          : usr=1.24%, sys=8.95%, ctx=2745, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16141: Fri Apr 19 03:39:46 2024
   read: IOPS=624, BW=2498KiB/s (2557kB/s)(5060KiB/2026msec)
    slat (usec): min=439, max=2233, avg=767.37, stdev=221.58
    clat (usec): min=3600, max=31488, avg=11946.11, stdev=3073.51
     lat (usec): min=4327, max=33416, avg=12715.20, stdev=3230.08
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 8848], 10.00th=[ 9372], 20.00th=[ 9896],
     | 30.00th=[10421], 40.00th=[10945], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13960], 95.00th=[17695],
     | 99.00th=[25560], 99.50th=[27132], 99.90th=[28443], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 1940, max= 2816, per=25.15%, avg=2489.00, stdev=380.41, samples=4
   iops        : min=  485, max=  704, avg=622.25, stdev=95.10, samples=4
  write: IOPS=639, BW=2557KiB/s (2618kB/s)(5180KiB/2026msec)
    slat (usec): min=12, max=698, avg=42.59, stdev=32.99
    clat (usec): min=2615, max=31195, avg=12396.36, stdev=3123.86
     lat (usec): min=2643, max=31313, avg=12439.96, stdev=3134.59
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10683],
     | 30.00th=[11076], 40.00th=[11469], 50.00th=[11731], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[14091], 95.00th=[19006],
     | 99.00th=[26870], 99.50th=[27919], 99.90th=[30016], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 2035, max= 2832, per=24.68%, avg=2542.75, stdev=349.36, samples=4
   iops        : min=  508, max=  708, avg=635.50, stdev=87.70, samples=4
  lat (msec)   : 4=0.08%, 10=14.14%, 20=81.41%, 50=4.38%
  cpu          : usr=0.99%, sys=9.23%, ctx=2684, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9895KiB/s (10.1MB/s), 2462KiB/s-2498KiB/s (2522kB/s-2557kB/s), io=19.6MiB (20.5MB), run=2017-2028msec
  WRITE: bw=10.1MiB/s (10.5MB/s), 2557KiB/s-2610KiB/s (2618kB/s-2672kB/s), io=20.4MiB (21.4MB), run=2017-2028msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16147: Fri Apr 19 03:39:48 2024
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=7253.0k, max=7253.0k, avg=7253952.00, stdev= 0.00
    clat (nsec): min=65110k, max=65110k, avg=65109617.00, stdev= 0.00
     lat (nsec): min=72377k, max=72377k, avg=72377399.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[65274],  5.00th=[65274], 10.00th=[65274], 20.00th=[65274],
     | 30.00th=[65274], 40.00th=[65274], 50.00th=[65274], 60.00th=[65274],
     | 70.00th=[65274], 80.00th=[65274], 90.00th=[65274], 95.00th=[65274],
     | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274],
     | 99.99th=[65274]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.33%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck