== sanity test 398c: run fio to test AIO ================= 13:41:03 (1713289263)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.132134 s, 317 MB/s
osc.lustre-OST0000-osc-ffff88012aa38000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=15716: Tue Apr 16 13:41:12 2024
  write: IOPS=362, BW=1449KiB/s (1483kB/s)(10.0MiB/7069msec)
    slat (usec): min=24, max=501, avg=81.40, stdev=40.71
    clat (usec): min=10231, max=82509, avg=43980.59, stdev=5407.68
     lat (usec): min=10263, max=82600, avg=44063.84, stdev=5408.41
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[32900], 10.00th=[36439], 20.00th=[41157],
     | 30.00th=[42730], 40.00th=[43779], 50.00th=[45876], 60.00th=[46400],
     | 70.00th=[47449], 80.00th=[47973], 90.00th=[48497], 95.00th=[49021],
     | 99.00th=[50594], 99.50th=[52691], 99.90th=[72877], 99.95th=[77071],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1328, max= 1688, per=25.02%, avg=1447.57, stdev=128.36, samples=14
   iops        : min=  332, max=  422, avg=361.86, stdev=32.11, samples=14
  lat (msec)   : 20=0.12%, 50=97.89%, 100=1.99%
  cpu          : usr=0.71%, sys=3.52%, ctx=1670, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=15717: Tue Apr 16 13:41:12 2024
  write: IOPS=362, BW=1450KiB/s (1485kB/s)(10.0MiB/7060msec)
    slat (usec): min=25, max=363, avg=85.26, stdev=41.66
    clat (usec): min=5371, max=82868, avg=43923.02, stdev=5568.82
     lat (usec): min=5607, max=82913, avg=44010.14, stdev=5567.84
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[33424], 10.00th=[36439], 20.00th=[41157],
     | 30.00th=[42730], 40.00th=[43779], 50.00th=[45876], 60.00th=[46400],
     | 70.00th=[47449], 80.00th=[47973], 90.00th=[48497], 95.00th=[49021],
     | 99.00th=[50070], 99.50th=[52691], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[83362]
   bw (  KiB/s): min= 1333, max= 1704, per=25.07%, avg=1450.36, stdev=126.58, samples=14
   iops        : min=  333, max=  426, avg=362.57, stdev=31.66, samples=14
  lat (msec)   : 10=0.20%, 20=0.16%, 50=98.01%, 100=1.64%
  cpu          : usr=0.52%, sys=3.78%, ctx=1872, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=15718: Tue Apr 16 13:41:12 2024
  write: IOPS=361, BW=1447KiB/s (1481kB/s)(10.0MiB/7078msec)
    slat (usec): min=22, max=506, avg=73.12, stdev=38.98
    clat (usec): min=26121, max=77441, avg=44030.67, stdev=5110.84
     lat (usec): min=26179, max=77532, avg=44105.54, stdev=5113.91
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[33817], 10.00th=[35914], 20.00th=[41157],
     | 30.00th=[42730], 40.00th=[43779], 50.00th=[45876], 60.00th=[46400],
     | 70.00th=[47449], 80.00th=[47973], 90.00th=[48497], 95.00th=[49021],
     | 99.00th=[50070], 99.50th=[50594], 99.90th=[67634], 99.95th=[72877],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1333, max= 1664, per=24.98%, avg=1445.21, stdev=122.78, samples=14
   iops        : min=  333, max=  416, avg=361.29, stdev=30.71, samples=14
  lat (msec)   : 50=98.67%, 100=1.33%
  cpu          : usr=0.59%, sys=3.12%, ctx=1176, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=15719: Tue Apr 16 13:41:12 2024
  write: IOPS=361, BW=1447KiB/s (1482kB/s)(10.0MiB/7077msec)
    slat (usec): min=21, max=839, avg=78.44, stdev=45.34
    clat (usec): min=5270, max=81830, avg=44036.27, stdev=5102.14
     lat (usec): min=5366, max=81877, avg=44116.43, stdev=5106.35
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[33817], 10.00th=[36439], 20.00th=[41157],
     | 30.00th=[42730], 40.00th=[43779], 50.00th=[45876], 60.00th=[46400],
     | 70.00th=[47449], 80.00th=[47973], 90.00th=[48497], 95.00th=[49021],
     | 99.00th=[50070], 99.50th=[50594], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1304, max= 1664, per=25.00%, avg=1446.21, stdev=118.12, samples=14
   iops        : min=  326, max=  416, avg=361.50, stdev=29.57, samples=14
  lat (msec)   : 10=0.04%, 50=99.02%, 100=0.94%
  cpu          : usr=0.52%, sys=3.46%, ctx=1575, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5787KiB/s (5926kB/s), 1447KiB/s-1450KiB/s (1481kB/s-1485kB/s), io=40.0MiB (41.9MB), run=7060-7078msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=15734: Tue Apr 16 13:41:16 2024
   read: IOPS=434, BW=1736KiB/s (1778kB/s)(5048KiB/2907msec)
    slat (usec): min=32, max=3884, avg=1304.66, stdev=380.79
    clat (usec): min=1408, max=39200, avg=17204.86, stdev=4471.62
     lat (usec): min=2199, max=41861, avg=18512.06, stdev=4669.18
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12518], 10.00th=[13304], 20.00th=[14091],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19006], 90.00th=[21103], 95.00th=[27132],
     | 99.00th=[34341], 99.50th=[36963], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1216, max= 1944, per=24.81%, avg=1712.00, stdev=289.94, samples=5
   iops        : min=  304, max=  486, avg=428.00, stdev=72.48, samples=5
  write: IOPS=446, BW=1786KiB/s (1829kB/s)(5192KiB/2907msec)
    slat (usec): min=12, max=1318, avg=58.45, stdev=54.72
    clat (usec): min=8965, max=40974, avg=17736.25, stdev=4378.04
     lat (usec): min=9905, max=41061, avg=17795.90, stdev=4382.53
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13960], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15795], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18744], 90.00th=[20841], 95.00th=[27657],
     | 99.00th=[36439], 99.50th=[38536], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1200, max= 2080, per=24.94%, avg=1792.00, stdev=351.77, samples=5
   iops        : min=  300, max=  520, avg=448.00, stdev=87.94, samples=5
  lat (msec)   : 2=0.04%, 10=0.39%, 20=86.33%, 50=13.24%
  cpu          : usr=0.96%, sys=9.46%, ctx=2585, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=15735: Tue Apr 16 13:41:16 2024
   read: IOPS=428, BW=1713KiB/s (1755kB/s)(4976KiB/2904msec)
    slat (usec): min=31, max=3084, avg=1315.93, stdev=381.98
    clat (usec): min=5072, max=40466, avg=17231.22, stdev=4379.11
     lat (usec): min=5111, max=43264, avg=18549.79, stdev=4602.07
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12911], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[21103], 95.00th=[27132],
     | 99.00th=[34866], 99.50th=[36439], 99.90th=[40109], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1112, max= 1872, per=24.51%, avg=1691.20, stdev=327.00, samples=5
   iops        : min=  278, max=  468, avg=422.80, stdev=81.75, samples=5
  write: IOPS=453, BW=1813KiB/s (1856kB/s)(5264KiB/2904msec)
    slat (usec): min=13, max=383, avg=56.68, stdev=35.38
    clat (usec): min=3639, max=40685, avg=17671.80, stdev=4253.52
     lat (usec): min=3699, max=40722, avg=17729.81, stdev=4258.67
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13698], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[20841], 95.00th=[27132],
     | 99.00th=[35914], 99.50th=[38536], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1344, max= 2016, per=25.37%, avg=1822.40, stdev=286.07, samples=5
   iops        : min=  336, max=  504, avg=455.60, stdev=71.52, samples=5
  lat (msec)   : 4=0.04%, 10=0.20%, 20=86.56%, 50=13.20%
  cpu          : usr=0.79%, sys=9.54%, ctx=2676, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=15736: Tue Apr 16 13:41:16 2024
   read: IOPS=429, BW=1719KiB/s (1760kB/s)(4984KiB/2900msec)
    slat (usec): min=29, max=3560, avg=1315.84, stdev=389.18
    clat (usec): min=4966, max=37842, avg=17061.36, stdev=4181.66
     lat (usec): min=5018, max=40757, avg=18379.73, stdev=4423.08
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12649], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[20579], 95.00th=[26608],
     | 99.00th=[34341], 99.50th=[35914], 99.90th=[37487], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1240, max= 1944, per=24.67%, avg=1702.40, stdev=304.13, samples=5
   iops        : min=  310, max=  486, avg=425.60, stdev=76.03, samples=5
  write: IOPS=453, BW=1812KiB/s (1856kB/s)(5256KiB/2900msec)
    slat (usec): min=12, max=903, avg=58.23, stdev=44.48
    clat (usec): min=9983, max=39870, avg=17783.99, stdev=4299.17
     lat (usec): min=10025, max=39961, avg=17843.57, stdev=4305.90
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[13960], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[20579], 95.00th=[28443],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[38536], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1224, max= 2000, per=25.21%, avg=1811.20, stdev=332.82, samples=5
   iops        : min=  306, max=  500, avg=452.80, stdev=83.21, samples=5
  lat (msec)   : 10=0.23%, 20=87.85%, 50=11.91%
  cpu          : usr=1.07%, sys=9.42%, ctx=2625, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=15737: Tue Apr 16 13:41:16 2024
   read: IOPS=435, BW=1740KiB/s (1782kB/s)(5060KiB/2908msec)
    slat (usec): min=34, max=3264, avg=1321.63, stdev=381.09
    clat (usec): min=5062, max=39868, avg=17154.38, stdev=4230.22
     lat (usec): min=5111, max=42029, avg=18478.51, stdev=4458.61
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12780], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18744], 90.00th=[20579], 95.00th=[26608],
     | 99.00th=[34866], 99.50th=[36439], 99.90th=[38011], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1224, max= 1944, per=24.97%, avg=1723.20, stdev=290.58, samples=5
   iops        : min=  306, max=  486, avg=430.80, stdev=72.64, samples=5
  write: IOPS=445, BW=1781KiB/s (1824kB/s)(5180KiB/2908msec)
    slat (usec): min=13, max=1318, avg=58.52, stdev=56.20
    clat (usec): min=11412, max=41004, avg=17775.76, stdev=4269.05
     lat (usec): min=11436, max=41073, avg=17835.42, stdev=4274.43
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13829], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[20579], 95.00th=[28181],
     | 99.00th=[36439], 99.50th=[38536], 99.90th=[38536], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1224, max= 1984, per=24.79%, avg=1780.80, stdev=316.31, samples=5
   iops        : min=  306, max=  496, avg=445.20, stdev=79.08, samples=5
  lat (msec)   : 10=0.23%, 20=87.42%, 50=12.34%
  cpu          : usr=1.00%, sys=9.32%, ctx=2632, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6901KiB/s (7067kB/s), 1713KiB/s-1740KiB/s (1755kB/s-1782kB/s), io=19.6MiB (20.5MB), run=2900-2908msec
  WRITE: bw=7184KiB/s (7357kB/s), 1781KiB/s-1813KiB/s (1824kB/s-1856kB/s), io=20.4MiB (21.4MB), run=2900-2908msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=15743: Tue Apr 16 13:41:18 2024
   read: IOPS=9, BW=367MiB/s (385MB/s)(40.0MiB/109msec)
    slat (nsec): min=6395.3k, max=6395.3k, avg=6395284.00, stdev= 0.00
    clat (nsec): min=101946k, max=101946k, avg=101946284.00, stdev= 0.00
     lat (nsec): min=108352k, max=108352k, avg=108351817.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  103],  5.00th=[  103], 10.00th=[  103], 20.00th=[  103],
     | 30.00th=[  103], 40.00th=[  103], 50.00th=[  103], 60.00th=[  103],
     | 70.00th=[  103], 80.00th=[  103], 90.00th=[  103], 95.00th=[  103],
     | 99.00th=[  103], 99.50th=[  103], 99.90th=[  103], 99.95th=[  103],
     | 99.99th=[  103]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.56%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=367MiB/s (385MB/s), 367MiB/s-367MiB/s (385MB/s-385MB/s), io=40.0MiB (41.9MB), run=109-109msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout