== sanity test 398c: run fio to test AIO ================= 09:33:23 (1713447203)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.148356 s, 283 MB/s
osc.lustre-OST0000-osc-ffff8800afed3800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18504: Thu Apr 18 09:33:32 2024
  write: IOPS=368, BW=1473KiB/s (1509kB/s)(10.0MiB/6950msec)
    slat (usec): min=21, max=1444, avg=82.22, stdev=58.27
    clat (usec): min=6930, max=86812, avg=43182.98, stdev=5862.35
     lat (usec): min=7047, max=86996, avg=43267.03, stdev=5862.88
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[31851], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[44827], 60.00th=[45351],
     | 70.00th=[46400], 80.00th=[46924], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[50070], 99.50th=[51643], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1344, max= 1848, per=25.02%, avg=1473.62, stdev=127.97, samples=13
   iops        : min=  336, max=  462, avg=368.31, stdev=32.03, samples=13
  lat (msec)   : 10=0.04%, 20=0.31%, 50=98.91%, 100=0.74%
  cpu          : usr=0.73%, sys=3.71%, ctx=1718, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18505: Thu Apr 18 09:33:32 2024
  write: IOPS=369, BW=1478KiB/s (1514kB/s)(10.0MiB/6927msec)
    slat (usec): min=20, max=1039, avg=82.41, stdev=48.64
    clat (usec): min=4857, max=86708, avg=43054.56, stdev=6347.49
     lat (usec): min=4976, max=86805, avg=43138.69, stdev=6348.51
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[28967], 10.00th=[38011], 20.00th=[41157],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[44827], 60.00th=[45351],
     | 70.00th=[45876], 80.00th=[46924], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[50070], 99.50th=[54789], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1360, max= 1936, per=25.16%, avg=1481.54, stdev=149.07, samples=13
   iops        : min=  340, max=  484, avg=370.38, stdev=37.27, samples=13
  lat (msec)   : 10=0.12%, 20=1.02%, 50=97.97%, 100=0.90%
  cpu          : usr=0.71%, sys=3.62%, ctx=1672, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18506: Thu Apr 18 09:33:32 2024
  write: IOPS=368, BW=1472KiB/s (1508kB/s)(10.0MiB/6955msec)
    slat (usec): min=22, max=1676, avg=81.33, stdev=71.27
    clat (usec): min=10849, max=90573, avg=43211.55, stdev=5724.74
     lat (usec): min=10898, max=90729, avg=43294.64, stdev=5722.85
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[31851], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[44827], 60.00th=[45351],
     | 70.00th=[46400], 80.00th=[46924], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[50070], 99.50th=[54264], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[90702]
   bw (  KiB/s): min= 1360, max= 1880, per=25.04%, avg=1474.85, stdev=134.72, samples=13
   iops        : min=  340, max=  470, avg=368.69, stdev=33.68, samples=13
  lat (msec)   : 20=0.51%, 50=98.44%, 100=1.05%
  cpu          : usr=0.53%, sys=3.67%, ctx=1479, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18507: Thu Apr 18 09:33:32 2024
  write: IOPS=368, BW=1473KiB/s (1508kB/s)(10.0MiB/6954msec)
    slat (usec): min=22, max=1291, avg=78.87, stdev=52.71
    clat (usec): min=6179, max=82485, avg=43209.38, stdev=5694.80
     lat (usec): min=6300, max=82583, avg=43290.04, stdev=5696.69
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[31851], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44827], 60.00th=[45351],
     | 70.00th=[46400], 80.00th=[46924], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[49546], 99.50th=[54264], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1360, max= 1824, per=25.06%, avg=1475.69, stdev=120.52, samples=13
   iops        : min=  340, max=  456, avg=368.92, stdev=30.13, samples=13
  lat (msec)   : 10=0.04%, 20=0.20%, 50=99.02%, 100=0.74%
  cpu          : usr=0.73%, sys=3.48%, ctx=1482, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5889KiB/s (6031kB/s), 1472KiB/s-1478KiB/s (1508kB/s-1514kB/s), io=40.0MiB (41.9MB), run=6927-6955msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18523: Thu Apr 18 09:33:37 2024
   read: IOPS=331, BW=1324KiB/s (1356kB/s)(5048KiB/3812msec)
    slat (usec): min=18, max=5172, avg=1627.20, stdev=424.12
    clat (usec): min=6456, max=41975, avg=22504.07, stdev=5042.83
     lat (usec): min=6586, max=44561, avg=24134.46, stdev=5283.78
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[14746], 10.00th=[16319], 20.00th=[18482],
     | 30.00th=[19792], 40.00th=[21103], 50.00th=[22414], 60.00th=[23462],
     | 70.00th=[24511], 80.00th=[26346], 90.00th=[28705], 95.00th=[31327],
     | 99.00th=[37487], 99.50th=[39584], 99.90th=[41681], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1064, max= 1408, per=24.16%, avg=1268.57, stdev=143.32, samples=7
   iops        : min=  266, max=  352, avg=317.14, stdev=35.83, samples=7
  write: IOPS=340, BW=1362KiB/s (1395kB/s)(5192KiB/3812msec)
    slat (usec): min=14, max=373, avg=72.19, stdev=40.77
    clat (usec): min=3979, max=41455, avg=23413.86, stdev=4559.88
     lat (usec): min=4013, max=41481, avg=23487.97, stdev=4565.12
    clat percentiles (usec):
     |  1.00th=[14484],  5.00th=[16188], 10.00th=[17695], 20.00th=[19792],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22938], 60.00th=[23987],
     | 70.00th=[25297], 80.00th=[26870], 90.00th=[29492], 95.00th=[31589],
     | 99.00th=[36439], 99.50th=[37487], 99.90th=[39584], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1008, max= 1480, per=24.27%, avg=1326.86, stdev=169.17, samples=7
   iops        : min=  252, max=  370, avg=331.71, stdev=42.29, samples=7
  lat (msec)   : 4=0.04%, 10=0.16%, 20=25.74%, 50=74.06%
  cpu          : usr=1.02%, sys=9.66%, ctx=2352, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18524: Thu Apr 18 09:33:37 2024
   read: IOPS=325, BW=1303KiB/s (1334kB/s)(4976KiB/3820msec)
    slat (usec): min=39, max=5469, avg=1661.39, stdev=425.04
    clat (usec): min=6427, max=43443, avg=22451.56, stdev=4986.50
     lat (usec): min=6507, max=46136, avg=24116.29, stdev=5208.46
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14746], 10.00th=[16188], 20.00th=[18220],
     | 30.00th=[19792], 40.00th=[21103], 50.00th=[22414], 60.00th=[23200],
     | 70.00th=[24511], 80.00th=[26084], 90.00th=[28967], 95.00th=[31851],
     | 99.00th=[36439], 99.50th=[38536], 99.90th=[41157], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  928, max= 1368, per=23.62%, avg=1240.00, stdev=172.57, samples=7
   iops        : min=  232, max=  342, avg=310.00, stdev=43.14, samples=7
  write: IOPS=344, BW=1378KiB/s (1411kB/s)(5264KiB/3820msec)
    slat (usec): min=12, max=1442, avg=72.60, stdev=57.31
    clat (usec): min=6302, max=44926, avg=23534.74, stdev=4673.00
     lat (usec): min=6453, max=44964, avg=23608.85, stdev=4677.38
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[16188], 10.00th=[17695], 20.00th=[19792],
     | 30.00th=[21103], 40.00th=[22152], 50.00th=[23200], 60.00th=[24249],
     | 70.00th=[25560], 80.00th=[27132], 90.00th=[29492], 95.00th=[31851],
     | 99.00th=[35914], 99.50th=[36963], 99.90th=[43779], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1112, max= 1520, per=24.44%, avg=1336.00, stdev=137.95, samples=7
   iops        : min=  278, max=  380, avg=334.00, stdev=34.49, samples=7
  lat (msec)   : 10=0.12%, 20=25.82%, 50=74.06%
  cpu          : usr=1.20%, sys=9.32%, ctx=2403, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18525: Thu Apr 18 09:33:37 2024
   read: IOPS=326, BW=1306KiB/s (1337kB/s)(4984KiB/3816msec)
    slat (usec): min=32, max=3476, avg=1642.26, stdev=414.13
    clat (usec): min=6405, max=45775, avg=22465.01, stdev=5002.58
     lat (usec): min=6444, max=48146, avg=24110.59, stdev=5228.69
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14877], 10.00th=[16450], 20.00th=[18482],
     | 30.00th=[19792], 40.00th=[21103], 50.00th=[22152], 60.00th=[23200],
     | 70.00th=[24511], 80.00th=[26084], 90.00th=[28443], 95.00th=[30802],
     | 99.00th=[36963], 99.50th=[39060], 99.90th=[45351], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min= 1016, max= 1432, per=23.99%, avg=1259.43, stdev=155.46, samples=7
   iops        : min=  254, max=  358, avg=314.86, stdev=38.86, samples=7
  write: IOPS=344, BW=1377KiB/s (1410kB/s)(5256KiB/3816msec)
    slat (usec): min=13, max=2297, avg=74.05, stdev=74.71
    clat (usec): min=11654, max=41791, avg=23479.57, stdev=4628.43
     lat (usec): min=11730, max=41850, avg=23555.30, stdev=4636.23
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[16057], 10.00th=[17695], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[22152], 50.00th=[23200], 60.00th=[24249],
     | 70.00th=[25297], 80.00th=[26870], 90.00th=[29492], 95.00th=[31589],
     | 99.00th=[36439], 99.50th=[38011], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1024, max= 1488, per=24.25%, avg=1325.71, stdev=159.18, samples=7
   iops        : min=  256, max=  372, avg=331.43, stdev=39.79, samples=7
  lat (msec)   : 10=0.12%, 20=25.62%, 50=74.26%
  cpu          : usr=1.05%, sys=9.57%, ctx=2392, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18526: Thu Apr 18 09:33:37 2024
   read: IOPS=330, BW=1324KiB/s (1356kB/s)(5060KiB/3822msec)
    slat (usec): min=37, max=3080, avg=1637.81, stdev=409.18
    clat (usec): min=8420, max=47024, avg=22686.86, stdev=5224.61
     lat (usec): min=10151, max=47062, avg=24327.95, stdev=5396.59
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14877], 10.00th=[16450], 20.00th=[18482],
     | 30.00th=[19792], 40.00th=[21365], 50.00th=[22414], 60.00th=[23462],
     | 70.00th=[24773], 80.00th=[26608], 90.00th=[29230], 95.00th=[31589],
     | 99.00th=[39584], 99.50th=[43254], 99.90th=[46400], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  960, max= 1400, per=24.23%, avg=1272.00, stdev=156.49, samples=7
   iops        : min=  240, max=  350, avg=318.00, stdev=39.12, samples=7
  write: IOPS=338, BW=1355KiB/s (1388kB/s)(5180KiB/3822msec)
    slat (usec): min=12, max=803, avg=75.01, stdev=49.27
    clat (usec): min=10157, max=38235, avg=23343.18, stdev=4683.24
     lat (usec): min=10211, max=38316, avg=23420.13, stdev=4690.37
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[15926], 10.00th=[17171], 20.00th=[19530],
     | 30.00th=[20841], 40.00th=[21890], 50.00th=[22938], 60.00th=[23987],
     | 70.00th=[25297], 80.00th=[26870], 90.00th=[29492], 95.00th=[31851],
     | 99.00th=[36963], 99.50th=[37487], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1000, max= 1448, per=23.73%, avg=1297.14, stdev=170.61, samples=7
   iops        : min=  250, max=  362, avg=324.29, stdev=42.65, samples=7
  lat (msec)   : 10=0.04%, 20=26.41%, 50=73.55%
  cpu          : usr=1.05%, sys=9.71%, ctx=2423, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5251KiB/s (5377kB/s), 1303KiB/s-1324KiB/s (1334kB/s-1356kB/s), io=19.6MiB (20.5MB), run=3812-3822msec
  WRITE: bw=5466KiB/s (5597kB/s), 1355KiB/s-1378KiB/s (1388kB/s-1411kB/s), io=20.4MiB (21.4MB), run=3812-3822msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18532: Thu Apr 18 09:33:38 2024
   read: IOPS=7, BW=282MiB/s (295MB/s)(40.0MiB/142msec)
    slat (nsec): min=6003.4k, max=6003.4k, avg=6003409.00, stdev= 0.00
    clat (nsec): min=135474k, max=135474k, avg=135474349.00, stdev= 0.00
     lat (nsec): min=141487k, max=141487k, avg=141486524.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  136],  5.00th=[  136], 10.00th=[  136], 20.00th=[  136],
     | 30.00th=[  136], 40.00th=[  136], 50.00th=[  136], 60.00th=[  136],
     | 70.00th=[  136], 80.00th=[  136], 90.00th=[  136], 95.00th=[  136],
     | 99.00th=[  136], 99.50th=[  136], 99.90th=[  136], 99.95th=[  136],
     | 99.99th=[  136]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.26%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=282MiB/s (295MB/s), 282MiB/s-282MiB/s (295MB/s-295MB/s), io=40.0MiB (41.9MB), run=142-142msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout