== sanity test 398c: run fio to test AIO ================= 12:54:00 (1713372840)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0840435 s, 499 MB/s
osc.lustre-OST0000-osc-ffff8800a926c800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=19801: Wed Apr 17 12:54:08 2024
  write: IOPS=392, BW=1569KiB/s (1607kB/s)(10.0MiB/6527msec)
    slat (usec): min=16, max=686, avg=83.88, stdev=45.36
    clat (usec): min=5497, max=74013, avg=40521.95, stdev=5777.10
     lat (usec): min=5587, max=74073, avg=40608.11, stdev=5783.85
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[28443], 10.00th=[32900], 20.00th=[36963],
     | 30.00th=[39584], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[43779], 80.00th=[44827], 90.00th=[45876], 95.00th=[47449],
     | 99.00th=[50070], 99.50th=[50594], 99.90th=[63177], 99.95th=[68682],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1408, max= 1912, per=24.97%, avg=1566.15, stdev=135.26, samples=13
   iops        : min=  352, max=  478, avg=391.54, stdev=33.81, samples=13
  lat (msec)   : 10=0.08%, 20=0.16%, 50=98.63%, 100=1.13%
  cpu          : usr=0.75%, sys=3.59%, ctx=1816, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19802: Wed Apr 17 12:54:08 2024
  write: IOPS=392, BW=1569KiB/s (1607kB/s)(10.0MiB/6527msec)
    slat (usec): min=13, max=629, avg=84.89, stdev=47.41
    clat (usec): min=5867, max=73993, avg=40519.56, stdev=5914.62
     lat (usec): min=5963, max=74096, avg=40606.75, stdev=5922.81
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[28443], 10.00th=[32900], 20.00th=[36963],
     | 30.00th=[39584], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[43779], 80.00th=[44827], 90.00th=[45876], 95.00th=[47449],
     | 99.00th=[50070], 99.50th=[52691], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1408, max= 1920, per=24.97%, avg=1566.15, stdev=137.37, samples=13
   iops        : min=  352, max=  480, avg=391.54, stdev=34.34, samples=13
  lat (msec)   : 10=0.12%, 20=0.04%, 50=98.67%, 100=1.17%
  cpu          : usr=0.63%, sys=3.66%, ctx=1899, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19803: Wed Apr 17 12:54:08 2024
  write: IOPS=392, BW=1570KiB/s (1608kB/s)(10.0MiB/6522msec)
    slat (usec): min=15, max=547, avg=79.92, stdev=43.71
    clat (usec): min=3939, max=70150, avg=40499.77, stdev=5904.67
     lat (usec): min=3998, max=70234, avg=40581.83, stdev=5908.47
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[28443], 10.00th=[32900], 20.00th=[36963],
     | 30.00th=[39060], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[43779], 80.00th=[44827], 90.00th=[45876], 95.00th=[47449],
     | 99.00th=[50594], 99.50th=[50594], 99.90th=[65274], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1408, max= 1912, per=24.97%, avg=1566.15, stdev=138.42, samples=13
   iops        : min=  352, max=  478, avg=391.54, stdev=34.60, samples=13
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.16%, 50=98.44%, 100=1.29%
  cpu          : usr=0.78%, sys=3.50%, ctx=1578, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19804: Wed Apr 17 12:54:08 2024
  write: IOPS=391, BW=1568KiB/s (1606kB/s)(10.0MiB/6531msec)
    slat (usec): min=14, max=794, avg=73.90, stdev=43.51
    clat (usec): min=3552, max=62895, avg=40558.76, stdev=5714.71
     lat (usec): min=3603, max=63009, avg=40634.74, stdev=5721.41
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[28443], 10.00th=[32900], 20.00th=[36963],
     | 30.00th=[39584], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[43779], 80.00th=[44827], 90.00th=[45876], 95.00th=[47449],
     | 99.00th=[49546], 99.50th=[51119], 99.90th=[62653], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1408, max= 1872, per=24.97%, avg=1566.15, stdev=120.49, samples=13
   iops        : min=  352, max=  468, avg=391.54, stdev=30.12, samples=13
  lat (msec)   : 4=0.04%, 10=0.12%, 20=0.04%, 50=98.83%, 100=0.98%
  cpu          : usr=0.54%, sys=3.40%, ctx=1276, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6272KiB/s (6422kB/s), 1568KiB/s-1570KiB/s (1606kB/s-1608kB/s), io=40.0MiB (41.9MB), run=6522-6531msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=19819: Wed Apr 17 12:54:11 2024
   read: IOPS=443, BW=1776KiB/s (1818kB/s)(5048KiB/2843msec)
    slat (usec): min=15, max=3473, avg=1201.53, stdev=340.46
    clat (usec): min=2359, max=41161, avg=16845.93, stdev=3926.08
     lat (usec): min=3797, max=43282, avg=18049.29, stdev=4078.91
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12387], 10.00th=[13173], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15664], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[20317], 95.00th=[23200],
     | 99.00th=[33817], 99.50th=[35914], 99.90th=[39060], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1400, max= 1940, per=24.90%, avg=1756.00, stdev=208.38, samples=5
   iops        : min=  350, max=  485, avg=439.00, stdev=52.10, samples=5
  write: IOPS=456, BW=1826KiB/s (1870kB/s)(5192KiB/2843msec)
    slat (usec): min=12, max=375, avg=42.55, stdev=23.95
    clat (usec): min=11339, max=39562, avg=17419.74, stdev=3831.51
     lat (usec): min=11406, max=39651, avg=17463.33, stdev=3838.58
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13698], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19792], 95.00th=[23462],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1448, max= 2099, per=25.12%, avg=1843.80, stdev=239.10, samples=5
   iops        : min=  362, max=  524, avg=460.80, stdev=59.58, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=89.57%, 50=10.31%
  cpu          : usr=0.67%, sys=6.93%, ctx=2641, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19820: Wed Apr 17 12:54:11 2024
   read: IOPS=437, BW=1750KiB/s (1792kB/s)(4976KiB/2844msec)
    slat (usec): min=14, max=3437, avg=1203.18, stdev=341.01
    clat (usec): min=1306, max=43886, avg=16851.78, stdev=4177.24
     lat (usec): min=2449, max=45840, avg=18056.96, stdev=4351.68
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12387], 10.00th=[13173], 20.00th=[14222],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16188], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[20317], 95.00th=[23462],
     | 99.00th=[33817], 99.50th=[39584], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1296, max= 1868, per=24.45%, avg=1724.00, stdev=242.36, samples=5
   iops        : min=  324, max=  467, avg=431.00, stdev=60.59, samples=5
  write: IOPS=462, BW=1851KiB/s (1895kB/s)(5264KiB/2844msec)
    slat (usec): min=13, max=226, avg=41.27, stdev=22.53
    clat (usec): min=10958, max=43084, avg=17441.74, stdev=3743.97
     lat (usec): min=10994, max=43124, avg=17484.01, stdev=3748.79
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13698], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19792], 95.00th=[22676],
     | 99.00th=[36963], 99.50th=[39060], 99.90th=[41157], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1560, max= 2059, per=25.36%, avg=1861.40, stdev=193.23, samples=5
   iops        : min=  390, max=  514, avg=465.20, stdev=48.12, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.20%, 20=89.80%, 50=9.92%
  cpu          : usr=0.60%, sys=6.89%, ctx=2612, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19821: Wed Apr 17 12:54:11 2024
   read: IOPS=439, BW=1757KiB/s (1799kB/s)(4984KiB/2837msec)
    slat (usec): min=13, max=3391, avg=1206.87, stdev=347.23
    clat (usec): min=1394, max=41139, avg=16634.93, stdev=3839.30
     lat (usec): min=2994, max=43281, avg=17843.56, stdev=4010.08
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[12518], 10.00th=[13173], 20.00th=[14091],
     | 30.00th=[14746], 40.00th=[15401], 50.00th=[16057], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[18482], 90.00th=[19792], 95.00th=[22676],
     | 99.00th=[33162], 99.50th=[35390], 99.90th=[40633], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1424, max= 2064, per=24.92%, avg=1756.80, stdev=234.37, samples=5
   iops        : min=  356, max=  516, avg=439.20, stdev=58.59, samples=5
  write: IOPS=463, BW=1853KiB/s (1897kB/s)(5256KiB/2837msec)
    slat (usec): min=12, max=1018, avg=42.34, stdev=33.76
    clat (usec): min=8038, max=40430, avg=17546.63, stdev=3964.09
     lat (usec): min=8051, max=40466, avg=17590.09, stdev=3969.05
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13566], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[20055], 95.00th=[25297],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[39584], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1432, max= 1992, per=25.26%, avg=1854.40, stdev=237.14, samples=5
   iops        : min=  358, max=  498, avg=463.60, stdev=59.29, samples=5
  lat (msec)   : 2=0.04%, 10=0.43%, 20=89.49%, 50=10.04%
  cpu          : usr=0.99%, sys=6.73%, ctx=2548, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19822: Wed Apr 17 12:54:11 2024
   read: IOPS=444, BW=1778KiB/s (1821kB/s)(5060KiB/2846msec)
    slat (usec): min=17, max=3773, avg=1204.73, stdev=346.00
    clat (usec): min=4637, max=37664, avg=16744.40, stdev=3890.89
     lat (usec): min=5560, max=41001, avg=17950.95, stdev=4065.07
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[12518], 10.00th=[13173], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15533], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19792], 95.00th=[22938],
     | 99.00th=[32375], 99.50th=[35390], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1408, max= 1952, per=25.03%, avg=1764.80, stdev=211.33, samples=5
   iops        : min=  352, max=  488, avg=441.20, stdev=52.83, samples=5
  write: IOPS=455, BW=1820KiB/s (1864kB/s)(5180KiB/2846msec)
    slat (usec): min=12, max=523, avg=44.41, stdev=31.84
    clat (usec): min=6212, max=44012, avg=17550.86, stdev=4087.36
     lat (usec): min=6240, max=44084, avg=17596.31, stdev=4097.70
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13829], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[20317], 95.00th=[22676],
     | 99.00th=[38011], 99.50th=[39060], 99.90th=[42206], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1400, max= 2064, per=24.72%, avg=1814.40, stdev=252.06, samples=5
   iops        : min=  350, max=  516, avg=453.60, stdev=63.01, samples=5
  lat (msec)   : 10=0.78%, 20=88.71%, 50=10.51%
  cpu          : usr=1.12%, sys=6.57%, ctx=2645, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7051KiB/s (7221kB/s), 1750KiB/s-1778KiB/s (1792kB/s-1821kB/s), io=19.6MiB (20.5MB), run=2837-2846msec
  WRITE: bw=7341KiB/s (7517kB/s), 1820KiB/s-1853KiB/s (1864kB/s-1897kB/s), io=20.4MiB (21.4MB), run=2837-2846msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=19828: Wed Apr 17 12:54:13 2024
   read: IOPS=11, BW=476MiB/s (499MB/s)(40.0MiB/84msec)
    slat (nsec): min=7880.5k, max=7880.5k, avg=7880476.00, stdev= 0.00
    clat (nsec): min=75268k, max=75268k, avg=75267795.00, stdev= 0.00
     lat (nsec): min=83163k, max=83163k, avg=83163165.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[74974],  5.00th=[74974], 10.00th=[74974], 20.00th=[74974],
     | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974],
     | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974],
     | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.64%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=476MiB/s (499MB/s), 476MiB/s-476MiB/s (499MB/s-499MB/s), io=40.0MiB (41.9MB), run=84-84msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout