== sanity test 398c: run fio to test AIO ================= 23:44:54 (1713411894)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0738094 s, 568 MB/s
osc.lustre-OST0000-osc-ffff88012a451000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16729: Wed Apr 17 23:44:59 2024
  write: IOPS=772, BW=3092KiB/s (3166kB/s)(10.0MiB/3312msec)
    slat (usec): min=10, max=296, avg=43.79, stdev=25.03
    clat (usec): min=2408, max=36661, avg=20556.52, stdev=2345.61
     lat (usec): min=2460, max=36704, avg=20601.38, stdev=2347.83
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[16712], 10.00th=[18220], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20579], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22676], 95.00th=[23200],
     | 99.00th=[25035], 99.50th=[25822], 99.90th=[34341], 99.95th=[34341],
     | 99.99th=[36439]
   bw (  KiB/s): min= 2944, max= 3344, per=25.01%, avg=3093.33, stdev=141.91, samples=6
   iops        : min=  736, max=  836, avg=773.33, stdev=35.48, samples=6
  lat (msec)   : 4=0.04%, 10=0.12%, 20=30.31%, 50=69.53%
  cpu          : usr=0.69%, sys=3.93%, ctx=1929, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16730: Wed Apr 17 23:44:59 2024
  write: IOPS=772, BW=3092KiB/s (3166kB/s)(10.0MiB/3312msec)
    slat (usec): min=10, max=259, avg=38.37, stdev=22.90
    clat (usec): min=11398, max=25848, avg=20564.40, stdev=2221.45
     lat (usec): min=11415, max=25885, avg=20603.78, stdev=2223.07
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[16712], 10.00th=[18220], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20579], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22676], 95.00th=[23200],
     | 99.00th=[24511], 99.50th=[25035], 99.90th=[25560], 99.95th=[25560],
     | 99.99th=[25822]
   bw (  KiB/s): min= 2944, max= 3328, per=24.98%, avg=3089.33, stdev=140.80, samples=6
   iops        : min=  736, max=  832, avg=772.33, stdev=35.20, samples=6
  lat (msec)   : 20=30.16%, 50=69.84%
  cpu          : usr=0.51%, sys=3.50%, ctx=1369, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16731: Wed Apr 17 23:44:59 2024
  write: IOPS=773, BW=3094KiB/s (3168kB/s)(10.0MiB/3310msec)
    slat (usec): min=11, max=273, avg=40.42, stdev=22.39
    clat (usec): min=2313, max=36453, avg=20548.23, stdev=2414.93
     lat (usec): min=2358, max=36480, avg=20589.68, stdev=2416.22
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[16450], 10.00th=[18220], 20.00th=[19268],
     | 30.00th=[19792], 40.00th=[20579], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22676], 95.00th=[23200],
     | 99.00th=[24511], 99.50th=[25560], 99.90th=[36439], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 2944, max= 3360, per=25.02%, avg=3094.67, stdev=148.76, samples=6
   iops        : min=  736, max=  840, avg=773.67, stdev=37.19, samples=6
  lat (msec)   : 4=0.08%, 10=0.12%, 20=30.74%, 50=69.06%
  cpu          : usr=0.66%, sys=3.75%, ctx=1647, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16732: Wed Apr 17 23:44:59 2024
  write: IOPS=774, BW=3099KiB/s (3174kB/s)(10.0MiB/3304msec)
    slat (usec): min=11, max=202, avg=41.01, stdev=23.32
    clat (usec): min=3154, max=41143, avg=20518.57, stdev=2758.23
     lat (usec): min=3210, max=41159, avg=20560.66, stdev=2760.10
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[16057], 10.00th=[18220], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20579], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22676], 95.00th=[22938],
     | 99.00th=[25035], 99.50th=[25822], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 2968, max= 3440, per=25.13%, avg=3108.00, stdev=172.97, samples=6
   iops        : min=  742, max=  860, avg=777.00, stdev=43.24, samples=6
  lat (msec)   : 4=0.08%, 10=1.05%, 20=29.45%, 50=69.41%
  cpu          : usr=0.64%, sys=3.63%, ctx=1518, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.1MiB/s (12.7MB/s), 3092KiB/s-3099KiB/s (3166kB/s-3174kB/s), io=40.0MiB (41.9MB), run=3304-3312msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16745: Wed Apr 17 23:45:02 2024
   read: IOPS=485, BW=1943KiB/s (1990kB/s)(5048KiB/2598msec)
    slat (usec): min=20, max=2253, avg=1103.50, stdev=222.73
    clat (usec): min=1976, max=25568, avg=15388.46, stdev=2031.23
     lat (usec): min=2012, max=26629, avg=16493.57, stdev=2035.09
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12256], 10.00th=[12780], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15401], 60.00th=[16057],
     | 70.00th=[16450], 80.00th=[17171], 90.00th=[17957], 95.00th=[18482],
     | 99.00th=[19530], 99.50th=[20055], 99.90th=[24249], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 1856, max= 2032, per=25.23%, avg=1947.20, stdev=63.90, samples=5
   iops        : min=  464, max=  508, avg=486.80, stdev=15.97, samples=5
  write: IOPS=499, BW=1998KiB/s (2046kB/s)(5192KiB/2598msec)
    slat (usec): min=11, max=933, avg=38.61, stdev=30.88
    clat (usec): min=3097, max=24302, avg=15916.05, stdev=1638.67
     lat (usec): min=3126, max=24336, avg=15955.67, stdev=1637.81
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13435], 10.00th=[13960], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15401], 50.00th=[15926], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[19792], 99.50th=[20317], 99.90th=[23462], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 1848, max= 2128, per=24.85%, avg=1996.80, stdev=105.16, samples=5
   iops        : min=  462, max=  532, avg=499.20, stdev=26.29, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.12%, 20=99.26%, 50=0.55%
  cpu          : usr=1.23%, sys=6.85%, ctx=2167, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16746: Wed Apr 17 23:45:02 2024
   read: IOPS=479, BW=1918KiB/s (1964kB/s)(4976KiB/2595msec)
    slat (usec): min=14, max=1878, avg=1093.62, stdev=209.45
    clat (usec): min=8924, max=26199, avg=15394.27, stdev=1955.68
     lat (usec): min=8982, max=27125, avg=16489.50, stdev=1967.20
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12387], 10.00th=[13042], 20.00th=[13829],
     | 30.00th=[14353], 40.00th=[14877], 50.00th=[15401], 60.00th=[15795],
     | 70.00th=[16319], 80.00th=[16909], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[19792], 99.50th=[20579], 99.90th=[23200], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 1816, max= 2000, per=24.75%, avg=1910.40, stdev=73.63, samples=5
   iops        : min=  454, max=  500, avg=477.60, stdev=18.41, samples=5
  write: IOPS=507, BW=2029KiB/s (2077kB/s)(5264KiB/2595msec)
    slat (usec): min=11, max=223, avg=37.73, stdev=17.57
    clat (usec): min=6983, max=26164, avg=15899.47, stdev=1557.67
     lat (usec): min=7005, max=26193, avg=15938.17, stdev=1557.78
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13435], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[15926], 60.00th=[16319],
     | 70.00th=[16712], 80.00th=[17171], 90.00th=[17695], 95.00th=[18220],
     | 99.00th=[19268], 99.50th=[19530], 99.90th=[21627], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 1904, max= 2168, per=25.29%, avg=2032.00, stdev=114.96, samples=5
   iops        : min=  476, max=  542, avg=508.00, stdev=28.74, samples=5
  lat (msec)   : 10=0.31%, 20=99.02%, 50=0.66%
  cpu          : usr=0.39%, sys=7.52%, ctx=2118, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16747: Wed Apr 17 23:45:02 2024
   read: IOPS=480, BW=1921KiB/s (1967kB/s)(4984KiB/2595msec)
    slat (usec): min=13, max=1877, avg=1102.84, stdev=216.41
    clat (usec): min=2038, max=21840, avg=15439.92, stdev=2094.37
     lat (usec): min=2082, max=22738, avg=16544.52, stdev=2103.56
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12125], 10.00th=[12780], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15401], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[20055], 99.50th=[20579], 99.90th=[21103], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min= 1872, max= 1984, per=25.00%, avg=1929.60, stdev=47.80, samples=5
   iops        : min=  468, max=  496, avg=482.40, stdev=11.95, samples=5
  write: IOPS=506, BW=2025KiB/s (2074kB/s)(5256KiB/2595msec)
    slat (usec): min=11, max=796, avg=38.61, stdev=26.94
    clat (usec): min=4359, max=25546, avg=15838.17, stdev=1690.25
     lat (usec): min=4441, max=25604, avg=15877.79, stdev=1688.79
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13304], 10.00th=[13829], 20.00th=[14484],
     | 30.00th=[15008], 40.00th=[15401], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17957], 95.00th=[18482],
     | 99.00th=[19792], 99.50th=[20579], 99.90th=[25560], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 1912, max= 2120, per=25.05%, avg=2012.80, stdev=75.39, samples=5
   iops        : min=  478, max=  530, avg=503.20, stdev=18.85, samples=5
  lat (msec)   : 4=0.04%, 10=0.23%, 20=98.75%, 50=0.98%
  cpu          : usr=0.93%, sys=7.17%, ctx=2115, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16748: Wed Apr 17 23:45:02 2024
   read: IOPS=486, BW=1946KiB/s (1993kB/s)(5060KiB/2600msec)
    slat (usec): min=15, max=2942, avg=1103.44, stdev=222.86
    clat (usec): min=3975, max=23245, avg=15496.23, stdev=2093.89
     lat (usec): min=4972, max=24440, avg=16601.35, stdev=2081.36
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12387], 10.00th=[12780], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[14877], 50.00th=[15401], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[21103], 99.50th=[21627], 99.90th=[22938], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 1864, max= 2008, per=25.08%, avg=1936.00, stdev=68.12, samples=5
   iops        : min=  466, max=  502, avg=484.00, stdev=17.03, samples=5
  write: IOPS=498, BW=1992KiB/s (2040kB/s)(5180KiB/2600msec)
    slat (usec): min=11, max=451, avg=37.89, stdev=22.48
    clat (usec): min=2316, max=20909, avg=15836.25, stdev=1573.43
     lat (usec): min=2357, max=20952, avg=15875.04, stdev=1572.75
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13435], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15139], 40.00th=[15533], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16581], 80.00th=[16909], 90.00th=[17695], 95.00th=[18482],
     | 99.00th=[19530], 99.50th=[20055], 99.90th=[20579], 99.95th=[20841],
     | 99.99th=[20841]
   bw (  KiB/s): min= 1912, max= 2048, per=24.71%, avg=1985.60, stdev=60.24, samples=5
   iops        : min=  478, max=  512, avg=496.40, stdev=15.06, samples=5
  lat (msec)   : 4=0.08%, 10=0.27%, 20=98.40%, 50=1.25%
  cpu          : usr=0.77%, sys=7.04%, ctx=2180, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7718KiB/s (7904kB/s), 1918KiB/s-1946KiB/s (1964kB/s-1993kB/s), io=19.6MiB (20.5MB), run=2595-2600msec
  WRITE: bw=8035KiB/s (8228kB/s), 1992KiB/s-2029KiB/s (2040kB/s-2077kB/s), io=20.4MiB (21.4MB), run=2595-2600msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16754: Wed Apr 17 23:45:03 2024
   read: IOPS=15, BW=625MiB/s (655MB/s)(40.0MiB/64msec)
    slat (nsec): min=6596.3k, max=6596.3k, avg=6596261.00, stdev= 0.00
    clat (nsec): min=56997k, max=56997k, avg=56997044.00, stdev= 0.00
     lat (nsec): min=63603k, max=63603k, avg=63602556.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[56886],  5.00th=[56886], 10.00th=[56886], 20.00th=[56886],
     | 30.00th=[56886], 40.00th=[56886], 50.00th=[56886], 60.00th=[56886],
     | 70.00th=[56886], 80.00th=[56886], 90.00th=[56886], 95.00th=[56886],
     | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886],
     | 99.99th=[56886]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.11%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=625MiB/s (655MB/s), 625MiB/s-625MiB/s (655MB/s-655MB/s), io=40.0MiB (41.9MB), run=64-64msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout