== sanity test 398b: DIO and buffer IO race ============== 18:27:30 (1713392850) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0755402 s, 167 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21211: Wed Apr 17 18:27:33 2024 read: IOPS=232, BW=931KiB/s (953kB/s)(1480KiB/1590msec) clat (usec): min=3, max=28709, avg=1145.61, stdev=2612.50 lat (usec): min=3, max=28710, avg=1146.21, stdev=2612.48 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 20], 60.00th=[ 709], | 70.00th=[ 1352], 80.00th=[ 1991], 90.00th=[ 2540], 95.00th=[ 3589], | 99.00th=[13304], 99.50th=[20841], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 736, max= 1104, per=24.88%, avg=909.33, stdev=184.93, samples=3 iops : min= 184, max= 276, avg=227.33, stdev=46.23, samples=3 write: IOPS=250, BW=1001KiB/s (1025kB/s)(1592KiB/1590msec) clat (usec): min=86, max=24316, avg=2910.70, stdev=4773.29 lat (usec): min=86, max=24316, avg=2911.44, stdev=4773.23 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 98], 10.00th=[ 104], 20.00th=[ 120], | 30.00th=[ 133], 40.00th=[ 188], 50.00th=[ 717], 60.00th=[ 1139], | 70.00th=[ 2008], 80.00th=[ 5080], 90.00th=[10552], 95.00th=[13173], | 99.00th=[20841], 99.50th=[22938], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 752, max= 1200, per=25.65%, avg=986.67, stdev=224.76, samples=3 iops : min= 188, max= 300, avg=246.67, stdev=56.19, samples=3 lat (usec) : 4=1.56%, 10=15.76%, 20=6.90%, 50=3.26%, 100=3.91% lat (usec) : 250=20.18%, 500=1.43%, 750=2.21%, 1000=5.47% lat (msec) : 2=14.19%, 4=11.85%, 10=6.77%, 20=5.34%, 50=1.17% cpu : usr=0.38%, sys=8.87%, ctx=835, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21212: Wed Apr 17 18:27:33 2024 read: IOPS=240, BW=964KiB/s (987kB/s)(1436KiB/1490msec) clat (usec): min=3, max=23796, avg=1045.65, stdev=2147.99 lat (usec): min=3, max=23797, avg=1046.50, stdev=2147.92 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 23], 60.00th=[ 848], | 70.00th=[ 1336], 80.00th=[ 1844], 90.00th=[ 2442], 95.00th=[ 3687], | 99.00th=[11338], 99.50th=[14353], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 928, max= 1024, per=26.70%, avg=976.00, stdev=67.88, samples=2 iops : min= 232, max= 256, avg=244.00, stdev=16.97, samples=2 write: IOPS=274, BW=1098KiB/s (1124kB/s)(1636KiB/1490msec) clat (usec): min=84, max=28897, avg=2702.48, stdev=4736.39 lat (usec): min=84, max=28898, avg=2703.06, stdev=4736.44 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 100], 10.00th=[ 108], 20.00th=[ 119], | 30.00th=[ 135], 40.00th=[ 178], 50.00th=[ 758], 60.00th=[ 1139], | 70.00th=[ 1909], 80.00th=[ 3556], 90.00th=[ 9372], 95.00th=[12387], | 99.00th=[22938], 99.50th=[23462], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 1088, max= 1088, per=28.29%, avg=1088.00, stdev= 0.00, samples=2 iops : min= 272, max= 272, avg=272.00, stdev= 0.00, samples=2 lat (usec) : 4=0.52%, 10=15.36%, 20=6.64%, 50=3.52%, 100=3.78% lat (usec) : 250=20.70%, 500=1.04%, 750=2.21%, 1000=6.77% lat (msec) : 2=16.28%, 4=11.33%, 10=6.64%, 20=4.17%, 50=1.04% cpu : usr=0.40%, sys=8.19%, ctx=769, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21213: Wed Apr 17 18:27:33 2024 read: IOPS=238, BW=955KiB/s (978kB/s)(1564KiB/1638msec) clat (usec): min=3, max=23147, avg=1292.60, stdev=2576.55 lat (usec): min=3, max=23148, avg=1293.11, stdev=2576.61 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 16], 50.00th=[ 30], 60.00th=[ 1172], | 70.00th=[ 1614], 80.00th=[ 2040], 90.00th=[ 2704], 95.00th=[ 3785], | 99.00th=[17695], 99.50th=[21890], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 616, max= 1168, per=24.30%, avg=888.00, stdev=276.09, samples=3 iops : min= 154, max= 292, avg=222.00, stdev=69.02, samples=3 write: IOPS=230, BW=921KiB/s (943kB/s)(1508KiB/1638msec) clat (usec): min=88, max=43786, avg=2982.78, stdev=5307.91 lat (usec): min=88, max=43787, avg=2983.34, stdev=5307.94 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 102], 10.00th=[ 110], 20.00th=[ 122], | 30.00th=[ 137], 40.00th=[ 190], 50.00th=[ 758], 60.00th=[ 1188], | 70.00th=[ 2024], 80.00th=[ 5014], 90.00th=[10552], 95.00th=[13304], | 99.00th=[23200], 99.50th=[36963], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 640, max= 1056, per=22.81%, avg=877.33, stdev=214.12, samples=3 iops : min= 160, max= 264, avg=219.33, stdev=53.53, samples=3 lat (usec) : 4=1.04%, 10=15.10%, 20=6.38%, 50=2.99%, 100=1.95% lat (usec) : 250=19.66%, 500=1.30%, 750=2.08%, 1000=6.12% lat (msec) : 2=17.97%, 4=12.89%, 10=6.38%, 20=5.08%, 50=1.04% cpu : usr=0.18%, sys=8.37%, ctx=860, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21214: Wed Apr 17 18:27:33 2024 read: IOPS=237, BW=951KiB/s (974kB/s)(1508KiB/1585msec) clat (usec): min=3, max=23697, avg=1024.33, stdev=2263.30 lat (usec): min=3, max=23698, avg=1024.81, stdev=2263.29 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 12], 50.00th=[ 20], 60.00th=[ 783], | 70.00th=[ 1336], 80.00th=[ 1926], 90.00th=[ 2409], 95.00th=[ 3064], | 99.00th=[15533], 99.50th=[17957], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 720, max= 1136, per=25.10%, avg=917.33, stdev=208.82, samples=3 iops : min= 180, max= 284, avg=229.33, stdev=52.20, samples=3 write: IOPS=246, BW=987KiB/s (1010kB/s)(1564KiB/1585msec) clat (usec): min=87, max=40438, avg=3043.20, stdev=5618.87 lat (usec): min=88, max=40439, avg=3043.76, stdev=5618.91 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 101], 10.00th=[ 109], 20.00th=[ 118], | 30.00th=[ 129], 40.00th=[ 169], 50.00th=[ 783], 60.00th=[ 1156], | 70.00th=[ 1827], 80.00th=[ 4490], 90.00th=[10552], 95.00th=[13173], | 99.00th=[30016], 99.50th=[39584], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 816, max= 1160, per=25.17%, avg=968.00, stdev=175.45, samples=3 iops : min= 204, max= 290, avg=242.00, stdev=43.86, samples=3 lat (usec) : 4=0.91%, 10=17.19%, 20=6.64%, 50=2.99%, 100=2.99% lat (usec) : 250=20.96%, 500=0.65%, 750=2.08%, 1000=6.38% lat (msec) : 2=16.67%, 4=10.16%, 10=5.99%, 20=5.08%, 50=1.30% cpu : usr=0.25%, sys=8.08%, ctx=784, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3656KiB/s (3743kB/s), 931KiB/s-964KiB/s (953kB/s-987kB/s), io=5988KiB (6132kB), run=1490-1638msec WRITE: bw=3846KiB/s (3938kB/s), 921KiB/s-1098KiB/s (943kB/s-1124kB/s), io=6300KiB (6451kB), run=1490-1638msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21206: Wed Apr 17 18:27:39 2024 read: IOPS=48, BW=194KiB/s (198kB/s)(1480KiB/7640msec) clat (usec): min=595, max=18796, avg=1444.98, stdev=1792.15 lat (usec): min=596, max=18797, avg=1445.59, stdev=1792.16 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 717], 10.00th=[ 758], 20.00th=[ 807], | 30.00th=[ 840], 40.00th=[ 898], 50.00th=[ 963], 60.00th=[ 1090], | 70.00th=[ 1221], 80.00th=[ 1450], 90.00th=[ 2114], 95.00th=[ 3458], | 99.00th=[11994], 99.50th=[14222], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 128, max= 296, per=25.45%, avg=190.87, stdev=51.55, samples=15 iops : min= 32, max= 74, avg=47.67, stdev=12.94, samples=15 write: IOPS=52, BW=208KiB/s (213kB/s)(1592KiB/7640msec) clat (usec): min=9985, max=42040, avg=17829.19, stdev=5207.26 lat (usec): min=9986, max=42042, avg=17829.92, stdev=5207.28 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[11731], 20.00th=[12780], | 30.00th=[14353], 40.00th=[16450], 50.00th=[18220], 60.00th=[19006], | 70.00th=[20317], 80.00th=[21365], 90.00th=[22414], 95.00th=[26608], | 99.00th=[38536], 99.50th=[41157], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 144, max= 248, per=26.29%, avg=207.40, stdev=30.72, samples=15 iops : min= 36, max= 62, avg=51.80, stdev= 7.66, samples=15 lat (usec) : 750=4.43%, 1000=21.09% lat (msec) : 2=17.58%, 4=3.12%, 10=1.43%, 20=35.68%, 50=16.67% cpu : usr=0.00%, sys=1.44%, ctx=803, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21207: Wed Apr 17 18:27:39 2024 read: IOPS=44, BW=180KiB/s (184kB/s)(1436KiB/7981msec) clat (usec): min=626, max=28198, avg=2032.32, stdev=3605.04 lat (usec): min=626, max=28199, avg=2032.94, stdev=3605.04 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 701], 10.00th=[ 742], 20.00th=[ 791], | 30.00th=[ 857], 40.00th=[ 922], 50.00th=[ 996], 60.00th=[ 1090], | 70.00th=[ 1254], 80.00th=[ 1549], 90.00th=[ 2671], 95.00th=[10814], | 99.00th=[20579], 99.50th=[21890], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 104, max= 256, per=23.82%, avg=178.67, stdev=42.73, samples=15 iops : min= 26, max= 64, avg=44.67, stdev=10.68, samples=15 write: IOPS=51, BW=205KiB/s (210kB/s)(1636KiB/7981msec) clat (usec): min=9829, max=46391, avg=17697.52, stdev=4724.69 lat (usec): min=9830, max=46392, avg=17698.61, stdev=4725.00 clat percentiles (usec): | 1.00th=[10290], 5.00th=[10814], 10.00th=[11469], 20.00th=[12518], | 30.00th=[14222], 40.00th=[16909], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20317], 80.00th=[21365], 90.00th=[22676], 95.00th=[24773], | 99.00th=[29754], 99.50th=[30540], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 144, max= 224, per=25.42%, avg=200.53, stdev=23.71, samples=15 iops : min= 36, max= 56, avg=50.13, stdev= 5.93, samples=15 lat (usec) : 750=5.34%, 1000=18.10% lat (msec) : 2=17.06%, 4=2.86%, 10=1.04%, 20=37.37%, 50=18.23% cpu : usr=0.09%, sys=1.30%, ctx=817, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21208: Wed Apr 17 18:27:39 2024 read: IOPS=51, BW=208KiB/s (213kB/s)(1564KiB/7526msec) clat (usec): min=607, max=23324, avg=1570.78, stdev=2471.09 lat (usec): min=608, max=23325, avg=1571.37, stdev=2471.09 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 693], 10.00th=[ 734], 20.00th=[ 775], | 30.00th=[ 824], 40.00th=[ 881], 50.00th=[ 963], 60.00th=[ 1074], | 70.00th=[ 1221], 80.00th=[ 1467], 90.00th=[ 1942], 95.00th=[ 2933], | 99.00th=[15270], 99.50th=[17957], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 104, max= 360, per=27.80%, avg=208.47, stdev=78.35, samples=15 iops : min= 26, max= 90, avg=52.07, stdev=19.59, samples=15 write: IOPS=50, BW=200KiB/s (205kB/s)(1508KiB/7526msec) clat (usec): min=10099, max=53186, avg=18305.23, stdev=5440.68 lat (usec): min=10100, max=53187, avg=18305.95, stdev=5440.69 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11469], 10.00th=[11994], 20.00th=[13173], | 30.00th=[15008], 40.00th=[16909], 50.00th=[18744], 60.00th=[19530], | 70.00th=[20579], 80.00th=[21365], 90.00th=[22676], 95.00th=[25560], | 99.00th=[41157], 99.50th=[43779], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 128, max= 232, per=25.41%, avg=200.47, stdev=27.38, samples=15 iops : min= 32, max= 58, avg=50.07, stdev= 6.79, samples=15 lat (usec) : 750=6.64%, 1000=21.09% lat (msec) : 2=18.49%, 4=2.73%, 10=0.39%, 20=33.20%, 50=17.32% lat (msec) : 100=0.13% cpu : usr=0.04%, sys=1.42%, ctx=806, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21209: Wed Apr 17 18:27:39 2024 read: IOPS=47, BW=192KiB/s (196kB/s)(1508KiB/7860msec) clat (usec): min=589, max=37884, avg=1934.60, stdev=4024.16 lat (usec): min=590, max=37885, avg=1935.25, stdev=4024.14 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 693], 10.00th=[ 725], 20.00th=[ 791], | 30.00th=[ 857], 40.00th=[ 914], 50.00th=[ 996], 60.00th=[ 1139], | 70.00th=[ 1287], 80.00th=[ 1549], 90.00th=[ 2073], 95.00th=[ 5669], | 99.00th=[24511], 99.50th=[33817], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 112, max= 280, per=24.88%, avg=186.60, stdev=50.19, samples=15 iops : min= 28, max= 70, avg=46.60, stdev=12.53, samples=15 write: IOPS=49, BW=199KiB/s (204kB/s)(1564KiB/7860msec) clat (usec): min=9930, max=40688, avg=18212.81, stdev=5460.45 lat (usec): min=9930, max=40690, avg=18213.57, stdev=5460.46 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11076], 10.00th=[11600], 20.00th=[13042], | 30.00th=[14877], 40.00th=[17171], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20579], 80.00th=[21365], 90.00th=[23462], 95.00th=[27395], | 99.00th=[39060], 99.50th=[40109], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 128, max= 224, per=24.87%, avg=196.20, stdev=31.78, samples=15 iops : min= 32, max= 56, avg=49.00, stdev= 7.90, samples=15 lat (usec) : 750=5.73%, 1000=18.88% lat (msec) : 2=19.40%, 4=2.08%, 10=1.43%, 20=33.85%, 50=18.62% cpu : usr=0.10%, sys=1.31%, ctx=808, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=750KiB/s (768kB/s), 180KiB/s-208KiB/s (184kB/s-213kB/s), io=5988KiB (6132kB), run=7526-7981msec WRITE: bw=789KiB/s (808kB/s), 199KiB/s-208KiB/s (204kB/s-213kB/s), io=6300KiB (6451kB), run=7526-7981msec