== sanity test 398c: run fio to test AIO ================= 23:34:20 (1713411260)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.131508 s, 319 MB/s
osc.lustre-OST0000-osc-ffff88008ff18800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=12555: Wed Apr 17 23:34:28 2024
  write: IOPS=387, BW=1549KiB/s (1586kB/s)(10.0MiB/6610msec)
    slat (usec): min=17, max=394, avg=74.07, stdev=40.15
    clat (usec): min=15375, max=63214, avg=41065.00, stdev=7783.19
     lat (usec): min=15428, max=63263, avg=41140.94, stdev=7785.96
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[25035], 10.00th=[29230], 20.00th=[33817],
     | 30.00th=[38011], 40.00th=[41681], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[46400], 80.00th=[47449], 90.00th=[47973], 95.00th=[49021],
     | 99.00th=[50594], 99.50th=[50594], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1333, max= 2304, per=24.96%, avg=1545.31, stdev=275.83, samples=13
   iops        : min=  333, max=  576, avg=386.31, stdev=68.97, samples=13
  lat (msec)   : 20=1.48%, 50=95.27%, 100=3.24%
  cpu          : usr=0.50%, sys=3.77%, ctx=1354, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=12556: Wed Apr 17 23:34:28 2024
  write: IOPS=388, BW=1553KiB/s (1590kB/s)(10.0MiB/6595msec)
    slat (usec): min=13, max=824, avg=77.62, stdev=45.69
    clat (usec): min=5621, max=79509, avg=40983.42, stdev=8240.41
     lat (usec): min=5763, max=79546, avg=41062.90, stdev=8238.25
    clat percentiles (usec):
     |  1.00th=[16712],  5.00th=[24511], 10.00th=[28443], 20.00th=[33817],
     | 30.00th=[38011], 40.00th=[41681], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[46400], 80.00th=[47449], 90.00th=[48497], 95.00th=[49546],
     | 99.00th=[50594], 99.50th=[51643], 99.90th=[69731], 99.95th=[74974],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1301, max= 2368, per=25.03%, avg=1549.62, stdev=298.01, samples=13
   iops        : min=  325, max=  592, avg=387.38, stdev=74.52, samples=13
  lat (msec)   : 10=0.16%, 20=1.48%, 50=94.92%, 100=3.44%
  cpu          : usr=0.74%, sys=3.52%, ctx=1520, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=12557: Wed Apr 17 23:34:28 2024
  write: IOPS=387, BW=1552KiB/s (1589kB/s)(10.0MiB/6600msec)
    slat (usec): min=14, max=619, avg=78.23, stdev=45.59
    clat (usec): min=5206, max=74955, avg=40993.66, stdev=8074.73
     lat (usec): min=5345, max=75053, avg=41073.93, stdev=8076.22
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[24773], 10.00th=[28705], 20.00th=[33817],
     | 30.00th=[38011], 40.00th=[41681], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[46400], 80.00th=[47449], 90.00th=[47973], 95.00th=[49546],
     | 99.00th=[50594], 99.50th=[50594], 99.90th=[66847], 99.95th=[69731],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1317, max= 2304, per=25.02%, avg=1549.00, stdev=277.61, samples=13
   iops        : min=  329, max=  576, avg=387.23, stdev=69.42, samples=13
  lat (msec)   : 10=0.20%, 20=1.52%, 50=95.16%, 100=3.12%
  cpu          : usr=0.67%, sys=3.61%, ctx=1563, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=12558: Wed Apr 17 23:34:28 2024
  write: IOPS=387, BW=1548KiB/s (1585kB/s)(10.0MiB/6614msec)
    slat (usec): min=14, max=409, avg=74.32, stdev=41.36
    clat (usec): min=16154, max=57936, avg=41086.94, stdev=7749.65
     lat (usec): min=16188, max=58019, avg=41163.11, stdev=7750.98
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[25297], 10.00th=[28967], 20.00th=[33817],
     | 30.00th=[38536], 40.00th=[41681], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[46400], 80.00th=[47449], 90.00th=[47973], 95.00th=[49546],
     | 99.00th=[50594], 99.50th=[51119], 99.90th=[53216], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1320, max= 2304, per=24.92%, avg=1542.77, stdev=279.14, samples=13
   iops        : min=  330, max=  576, avg=385.69, stdev=69.78, samples=13
  lat (msec)   : 20=1.48%, 50=95.23%, 100=3.28%
  cpu          : usr=0.62%, sys=3.42%, ctx=1445, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6193KiB/s (6342kB/s), 1548KiB/s-1553KiB/s (1585kB/s-1590kB/s), io=40.0MiB (41.9MB), run=6595-6614msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=12573: Wed Apr 17 23:34:33 2024
   read: IOPS=339, BW=1357KiB/s (1390kB/s)(5048KiB/3719msec)
    slat (usec): min=21, max=4318, avg=1506.75, stdev=444.41
    clat (usec): min=5248, max=47619, avg=21942.33, stdev=5728.21
     lat (usec): min=6617, max=47641, avg=23451.76, stdev=5916.73
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[14353], 10.00th=[15664], 20.00th=[17171],
     | 30.00th=[18482], 40.00th=[19792], 50.00th=[21103], 60.00th=[22414],
     | 70.00th=[23987], 80.00th=[26346], 90.00th=[28967], 95.00th=[32113],
     | 99.00th=[41157], 99.50th=[45351], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min=  920, max= 1712, per=24.89%, avg=1342.86, stdev=270.76, samples=7
   iops        : min=  230, max=  428, avg=335.71, stdev=67.69, samples=7
  write: IOPS=349, BW=1396KiB/s (1430kB/s)(5192KiB/3719msec)
    slat (usec): min=12, max=241, avg=51.85, stdev=28.88
    clat (usec): min=12317, max=42478, avg=22952.19, stdev=5513.76
     lat (usec): min=12370, max=42502, avg=23005.24, stdev=5518.49
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15533], 10.00th=[16581], 20.00th=[17957],
     | 30.00th=[19530], 40.00th=[20841], 50.00th=[22152], 60.00th=[23462],
     | 70.00th=[25297], 80.00th=[27657], 90.00th=[30540], 95.00th=[33817],
     | 99.00th=[38536], 99.50th=[40633], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  912, max= 1784, per=24.80%, avg=1393.14, stdev=286.68, samples=7
   iops        : min=  228, max=  446, avg=348.29, stdev=71.67, samples=7
  lat (msec)   : 10=0.04%, 20=37.27%, 50=62.70%
  cpu          : usr=1.05%, sys=6.62%, ctx=2741, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=12574: Wed Apr 17 23:34:33 2024
   read: IOPS=334, BW=1338KiB/s (1370kB/s)(4976KiB/3718msec)
    slat (usec): min=18, max=4735, avg=1524.78, stdev=433.55
    clat (usec): min=3839, max=45243, avg=21894.89, stdev=5438.86
     lat (usec): min=5065, max=45270, avg=23422.05, stdev=5668.01
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14484], 10.00th=[15664], 20.00th=[17171],
     | 30.00th=[18482], 40.00th=[19792], 50.00th=[21103], 60.00th=[22414],
     | 70.00th=[23987], 80.00th=[26346], 90.00th=[29230], 95.00th=[32375],
     | 99.00th=[36439], 99.50th=[38011], 99.90th=[43254], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min=  872, max= 1640, per=24.40%, avg=1316.57, stdev=256.70, samples=7
   iops        : min=  218, max=  410, avg=329.14, stdev=64.18, samples=7
  write: IOPS=353, BW=1416KiB/s (1450kB/s)(5264KiB/3718msec)
    slat (usec): min=13, max=327, avg=53.90, stdev=32.57
    clat (usec): min=7865, max=40499, avg=22964.22, stdev=5410.27
     lat (usec): min=7886, max=40518, avg=23019.34, stdev=5416.26
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15533], 10.00th=[16909], 20.00th=[18482],
     | 30.00th=[19530], 40.00th=[20841], 50.00th=[22152], 60.00th=[23462],
     | 70.00th=[25297], 80.00th=[27395], 90.00th=[30802], 95.00th=[33817],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min=  992, max= 1904, per=25.27%, avg=1419.43, stdev=302.13, samples=7
   iops        : min=  248, max=  476, avg=354.86, stdev=75.53, samples=7
  lat (msec)   : 4=0.04%, 10=0.08%, 20=36.88%, 50=63.01%
  cpu          : usr=1.08%, sys=6.73%, ctx=3020, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=12575: Wed Apr 17 23:34:33 2024
   read: IOPS=335, BW=1343KiB/s (1375kB/s)(4984KiB/3712msec)
    slat (usec): min=20, max=4218, avg=1521.62, stdev=427.83
    clat (usec): min=10204, max=43221, avg=21813.98, stdev=5295.03
     lat (usec): min=11341, max=43243, avg=23338.02, stdev=5513.68
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[14484], 10.00th=[15664], 20.00th=[17171],
     | 30.00th=[18482], 40.00th=[20055], 50.00th=[21103], 60.00th=[22414],
     | 70.00th=[23987], 80.00th=[26084], 90.00th=[28967], 95.00th=[31851],
     | 99.00th=[36439], 99.50th=[37487], 99.90th=[40633], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  912, max= 1752, per=24.89%, avg=1342.86, stdev=283.16, samples=7
   iops        : min=  228, max=  438, avg=335.71, stdev=70.79, samples=7
  write: IOPS=353, BW=1416KiB/s (1450kB/s)(5256KiB/3712msec)
    slat (usec): min=12, max=535, avg=54.84, stdev=34.79
    clat (usec): min=11638, max=42561, avg=22979.26, stdev=5554.42
     lat (usec): min=11689, max=42630, avg=23035.29, stdev=5559.42
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15533], 10.00th=[16319], 20.00th=[18220],
     | 30.00th=[19530], 40.00th=[20841], 50.00th=[22152], 60.00th=[23462],
     | 70.00th=[25035], 80.00th=[27395], 90.00th=[30802], 95.00th=[33817],
     | 99.00th=[38536], 99.50th=[39584], 99.90th=[40633], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  968, max= 1768, per=24.84%, avg=1395.43, stdev=271.13, samples=7
   iops        : min=  242, max=  442, avg=348.86, stdev=67.78, samples=7
  lat (msec)   : 20=37.03%, 50=62.97%
  cpu          : usr=0.73%, sys=6.95%, ctx=3036, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=12576: Wed Apr 17 23:34:33 2024
   read: IOPS=340, BW=1361KiB/s (1394kB/s)(5060KiB/3718msec)
    slat (usec): min=18, max=4313, avg=1520.69, stdev=428.07
    clat (usec): min=6328, max=47722, avg=21944.38, stdev=5451.55
     lat (usec): min=8189, max=47746, avg=23467.56, stdev=5649.59
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14353], 10.00th=[15795], 20.00th=[17433],
     | 30.00th=[18744], 40.00th=[19792], 50.00th=[21103], 60.00th=[22414],
     | 70.00th=[23987], 80.00th=[26346], 90.00th=[29230], 95.00th=[31327],
     | 99.00th=[38536], 99.50th=[39584], 99.90th=[47449], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min=  920, max= 1688, per=25.01%, avg=1349.71, stdev=276.50, samples=7
   iops        : min=  230, max=  422, avg=337.43, stdev=69.13, samples=7
  write: IOPS=348, BW=1393KiB/s (1427kB/s)(5180KiB/3718msec)
    slat (usec): min=12, max=479, avg=54.32, stdev=35.99
    clat (usec): min=12711, max=42535, avg=22911.87, stdev=5492.74
     lat (usec): min=12810, max=42555, avg=22967.37, stdev=5500.31
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15533], 10.00th=[16450], 20.00th=[17957],
     | 30.00th=[19530], 40.00th=[20841], 50.00th=[21890], 60.00th=[23200],
     | 70.00th=[25035], 80.00th=[27395], 90.00th=[30802], 95.00th=[33817],
     | 99.00th=[36963], 99.50th=[38536], 99.90th=[40633], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  960, max= 1808, per=24.62%, avg=1382.86, stdev=260.89, samples=7
   iops        : min=  240, max=  452, avg=345.71, stdev=65.22, samples=7
  lat (msec)   : 10=0.08%, 20=36.60%, 50=63.32%
  cpu          : usr=1.00%, sys=6.78%, ctx=2894, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5396KiB/s (5526kB/s), 1338KiB/s-1361KiB/s (1370kB/s-1394kB/s), io=19.6MiB (20.5MB), run=3712-3719msec
  WRITE: bw=5618KiB/s (5752kB/s), 1393KiB/s-1416KiB/s (1427kB/s-1450kB/s), io=20.4MiB (21.4MB), run=3712-3719msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=12583: Wed Apr 17 23:34:34 2024
   read: IOPS=7, BW=290MiB/s (304MB/s)(40.0MiB/138msec)
    slat (nsec): min=6041.7k, max=6041.7k, avg=6041651.00, stdev= 0.00
    clat (nsec): min=130934k, max=130934k, avg=130933853.00, stdev= 0.00
     lat (nsec): min=136984k, max=136984k, avg=136983884.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  131], 10.00th=[  131], 20.00th=[  131],
     | 30.00th=[  131], 40.00th=[  131], 50.00th=[  131], 60.00th=[  131],
     | 70.00th=[  131], 80.00th=[  131], 90.00th=[  131], 95.00th=[  131],
     | 99.00th=[  131], 99.50th=[  131], 99.90th=[  131], 99.95th=[  131],
     | 99.99th=[  131]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.38%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=40.0MiB (41.9MB), run=138-138msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout