== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:51:18 (1713531078) Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg150-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4635 (time 1713531116.58 total 10.00 last 463.44) - open/close 8317 (time 1713531126.59 total 20.00 last 368.14) - open/close 10000 (time 1713531131.82 total 25.24 last 321.39) - open/close 12952 (time 1713531141.82 total 35.24 last 295.17) - open/close 15141 (time 1713531151.83 total 45.25 last 218.76) - open/close 16805 (time 1713531161.83 total 55.25 last 166.38) - open/close 18447 (time 1713531171.83 total 65.25 last 164.18) - open/close 19884 (time 1713531181.84 total 75.25 last 143.61) - open/close 21202 (time 1713531191.84 total 85.26 last 131.70) - open/close 22645 (time 1713531201.85 total 95.26 last 144.28) - open/close 24107 (time 1713531211.85 total 105.27 last 146.13) - open/close 25451 (time 1713531221.85 total 115.27 last 134.39) - open/close 26676 (time 1713531231.86 total 125.28 last 122.43) - open/close 28019 (time 1713531241.86 total 135.28 last 134.22) - open/close 29351 (time 1713531251.87 total 145.28 last 133.16) - open/close 30000 (time 1713531256.65 total 150.07 last 135.69) - open/close 31264 (time 1713531266.65 total 160.07 last 126.36) - open/close 32410 (time 1713531276.66 total 170.08 last 114.48) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 171.44 seconds: 189.92 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1294 (time 1713531288.88 total 10.00 last 129.39) - /unlink 2502 (time 1713531298.88 total 20.00 last 120.79) - /unlink 3628 (time 1713531308.88 total 30.01 last 112.53) - /unlink 5107 (time 1713531318.88 total 40.01 last 147.89) - /unlink 6691 (time 1713531328.89 total 50.01 last 158.36) - /unlink 8382 (time 1713531338.89 total 60.01 last 169.05) - /unlink 10000 (time 1713531347.57 total 68.69 last 186.46) - /unlink 11765 (time 1713531357.57 total 78.69 last 176.47) - /unlink 13929 (time 1713531367.57 total 88.69 last 216.36) - /unlink 16287 (time 1713531377.57 total 98.70 last 235.73) - /unlink 19258 (time 1713531387.57 total 108.70 last 297.09) - /unlink 20000 (time 1713531389.85 total 110.98 last 325.71) - /unlink 23727 (time 1713531399.85 total 120.98 last 372.65) - /unlink 28801 (time 1713531409.85 total 130.98 last 507.37) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 135.74 seconds: 239.88 ops/second Waiting for MDT destroys to complete - open/close 4630 (time 1713531432.94 total 10.00 last 462.99) - open/close 8128 (time 1713531442.94 total 20.00 last 349.71) - open/close 10000 (time 1713531449.09 total 26.16 last 304.14) - open/close 12763 (time 1713531459.10 total 36.16 last 276.22) - open/close 15132 (time 1713531469.10 total 46.16 last 236.86) - open/close 17462 (time 1713531479.10 total 56.17 last 232.87) - open/close 19277 (time 1713531489.11 total 66.17 last 181.47) - open/close 20000 (time 1713531493.70 total 70.76 last 157.36) - open/close 21888 (time 1713531503.70 total 80.77 last 188.77) - open/close 23663 (time 1713531513.70 total 90.77 last 177.44) - open/close 25306 (time 1713531523.71 total 100.77 last 164.22) - open/close 26868 (time 1713531533.71 total 110.77 last 156.18) - open/close 28253 (time 1713531543.72 total 120.78 last 138.40) - open/close 29625 (time 1713531553.72 total 130.78 last 137.18) - open/close 30000 (time 1713531556.62 total 133.69 last 129.14) - open/close 31076 (time 1713531566.63 total 143.70 last 107.50) - open/close 32219 (time 1713531576.64 total 153.71 last 114.17) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 156.84 seconds: 207.59 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1253 (time 1713531590.65 total 10.01 last 125.19) - /unlink 2558 (time 1713531600.65 total 20.01 last 130.49) - /unlink 3881 (time 1713531610.65 total 30.01 last 132.26) - /unlink 5212 (time 1713531620.65 total 40.01 last 133.08) - /unlink 6661 (time 1713531630.66 total 50.02 last 144.80) - /unlink 8300 (time 1713531640.66 total 60.03 last 163.81) - /unlink 10000 (time 1713531650.04 total 69.40 last 181.31) - /unlink 11900 (time 1713531660.04 total 79.40 last 189.99) - /unlink 14120 (time 1713531670.05 total 89.41 last 221.85) - /unlink 16310 (time 1713531680.05 total 99.41 last 218.93) - /unlink 18902 (time 1713531690.05 total 109.41 last 259.17) - /unlink 20000 (time 1713531693.94 total 113.30 last 282.37) - /unlink 23497 (time 1713531703.94 total 123.30 last 349.68) - /unlink 28149 (time 1713531713.94 total 133.30 last 465.14) - /unlink 30000 (time 1713531716.69 total 136.05 last 673.81) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 139.11 seconds: 234.06 ops/second Waiting for MDT destroys to complete - open/close 4719 (time 1713531735.61 total 10.00 last 471.89) - open/close 8403 (time 1713531745.61 total 20.00 last 368.39) - open/close 10000 (time 1713531750.68 total 25.07 last 315.09) - open/close 12921 (time 1713531760.68 total 35.07 last 292.08) - open/close 15530 (time 1713531770.69 total 45.07 last 260.87) - open/close 17688 (time 1713531780.69 total 55.07 last 215.72) - open/close 19703 (time 1713531790.69 total 65.08 last 201.47) - open/close 21640 (time 1713531800.69 total 75.08 last 193.66) - open/close 23405 (time 1713531810.70 total 85.08 last 176.43) - open/close 24971 (time 1713531820.70 total 95.08 last 156.56) - open/close 26381 (time 1713531830.71 total 105.09 last 140.89) - open/close 27572 (time 1713531840.71 total 115.10 last 119.03) - open/close 28771 (time 1713531850.71 total 125.10 last 119.89) - open/close 30000 (time 1713531860.51 total 134.90 last 125.45) - open/close 31189 (time 1713531870.51 total 144.90 last 118.87) - open/close 32184 (time 1713531880.52 total 154.90 last 99.46) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 158.70 seconds: 205.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1187 (time 1713531895.50 total 10.00 last 118.65) - /unlink 2532 (time 1713531905.51 total 20.01 last 134.42) - /unlink 3908 (time 1713531915.51 total 30.02 last 137.51) - /unlink 5156 (time 1713531925.52 total 40.02 last 124.70) - /unlink 6568 (time 1713531935.52 total 50.02 last 141.19) - /unlink 8081 (time 1713531945.52 total 60.03 last 151.25) - /unlink 9598 (time 1713531955.53 total 70.03 last 151.66) - /unlink 10000 (time 1713531958.38 total 72.88 last 140.88) - /unlink 11296 (time 1713531968.38 total 82.89 last 129.57) - /unlink 13113 (time 1713531978.38 total 92.89 last 181.69) - /unlink 15000 (time 1713531988.39 total 102.89 last 188.65) - /unlink 17078 (time 1713531998.39 total 112.89 last 207.74) - /unlink 19457 (time 1713532008.39 total 122.89 last 237.87) - /unlink 20000 (time 1713532011.11 total 125.61 last 200.05) - /unlink 22707 (time 1713532021.11 total 135.61 last 270.69) - /unlink 26095 (time 1713532031.11 total 145.61 last 338.73) - /unlink 29349 (time 1713532041.11 total 155.61 last 325.38) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 161.68 seconds: 201.38 ops/second Waiting for MDT destroys to complete - open/close 3657 (time 1713532065.89 total 10.00 last 365.60) - open/close 5803 (time 1713532075.90 total 20.01 last 214.52) - open/close 7859 (time 1713532085.90 total 30.01 last 205.53) - open/close 9754 (time 1713532095.90 total 40.02 last 189.39) - open/close 11446 (time 1713532105.91 total 50.02 last 169.18) - open/close 13118 (time 1713532115.91 total 60.02 last 167.18) - open/close 14805 (time 1713532125.91 total 70.02 last 168.66) - open/close 16203 (time 1713532135.92 total 80.03 last 139.71) - open/close 17592 (time 1713532145.92 total 90.03 last 138.90) - open/close 18949 (time 1713532155.92 total 100.03 last 135.61) - open/close 20000 (time 1713532163.91 total 108.02 last 131.67) - open/close 21331 (time 1713532173.91 total 118.02 last 133.09) - open/close 22526 (time 1713532183.91 total 128.02 last 119.41) - open/close 23814 (time 1713532193.92 total 138.03 last 128.74) - open/close 24998 (time 1713532203.92 total 148.03 last 118.36) - open/close 26152 (time 1713532213.93 total 158.04 last 115.32) - open/close 27243 (time 1713532223.93 total 168.04 last 109.04) - open/close 28310 (time 1713532233.93 total 178.05 last 106.69) - open/close 29310 (time 1713532243.94 total 188.05 last 99.94) - open/close 30000 (time 1713532250.56 total 194.67 last 104.28) - open/close 31145 (time 1713532260.56 total 204.67 last 114.45) - open/close 32173 (time 1713532270.56 total 214.67 last 102.80) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 218.54 seconds: 148.99 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 999 (time 1713532285.75 total 10.01 last 99.84) - /unlink 2167 (time 1713532295.76 total 20.01 last 116.78) - /unlink 3347 (time 1713532305.76 total 30.01 last 117.99) - /unlink 4575 (time 1713532315.76 total 40.01 last 122.72) - /unlink 5875 (time 1713532325.77 total 50.02 last 129.91) - /unlink 7071 (time 1713532335.77 total 60.02 last 119.55) - /unlink 8282 (time 1713532345.78 total 70.03 last 121.08) - /unlink 9749 (time 1713532355.78 total 80.03 last 146.63) - /unlink 11061 (time 1713532365.78 total 90.04 last 131.13) - /unlink 12933 (time 1713532375.79 total 100.04 last 187.17) - /unlink 15155 (time 1713532385.79 total 110.04 last 222.10) - /unlink 17502 (time 1713532395.79 total 120.05 last 234.64) - /unlink 20000 (time 1713532404.86 total 129.11 last 275.47) - /unlink 22557 (time 1713532414.86 total 139.12 last 255.62) - /unlink 25730 (time 1713532424.87 total 149.12 last 317.23) - /unlink 29631 (time 1713532434.87 total 159.12 last 389.99) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 165.62 seconds: 196.60 ops/second Waiting for MDT destroys to complete - open/close 2970 (time 1713532462.52 total 10.00 last 296.93) - open/close 5040 (time 1713532472.52 total 20.01 last 206.91) - open/close 7057 (time 1713532482.52 total 30.01 last 201.64) - open/close 8860 (time 1713532492.53 total 40.01 last 180.22) - open/close 10000 (time 1713532499.10 total 46.59 last 173.38) - open/close 11650 (time 1713532509.11 total 56.59 last 164.91) - open/close 13226 (time 1713532519.11 total 66.60 last 157.57) - open/close 14741 (time 1713532529.11 total 76.60 last 151.46) - open/close 16219 (time 1713532539.12 total 86.60 last 147.77) - open/close 17628 (time 1713532549.12 total 96.60 last 140.90) - open/close 19010 (time 1713532559.12 total 106.61 last 138.14) - open/close 20000 (time 1713532566.59 total 114.07 last 132.57) - open/close 21313 (time 1713532576.59 total 124.08 last 131.26) - open/close 22695 (time 1713532586.60 total 134.08 last 138.14) - open/close 24106 (time 1713532596.60 total 144.08 last 141.07) - open/close 25411 (time 1713532606.61 total 154.09 last 130.40) - open/close 26763 (time 1713532616.61 total 164.09 last 135.16) - open/close 28002 (time 1713532626.61 total 174.10 last 123.87) - open/close 29431 (time 1713532636.61 total 184.10 last 142.90) - open/close 30000 (time 1713532641.77 total 189.26 last 110.28) - open/close 31062 (time 1713532651.77 total 199.26 last 106.17) - open/close 32122 (time 1713532661.78 total 209.26 last 105.98) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 213.42 seconds: 152.56 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1195 (time 1713532677.25 total 10.00 last 119.50) - /unlink 2719 (time 1713532687.26 total 20.01 last 152.32) - /unlink 4230 (time 1713532697.26 total 30.01 last 151.07) - /unlink 5925 (time 1713532707.26 total 40.01 last 169.47) - /unlink 7528 (time 1713532717.27 total 50.01 last 160.21) - /unlink 9044 (time 1713532727.27 total 60.02 last 151.54) - /unlink 10000 (time 1713532734.39 total 67.13 last 134.36) - /unlink 11491 (time 1713532744.39 total 77.14 last 149.09) - /unlink 13271 (time 1713532754.39 total 87.14 last 177.94) - /unlink 15564 (time 1713532764.39 total 97.14 last 229.24) - /unlink 17416 (time 1713532774.40 total 107.14 last 185.16) - /unlink 19184 (time 1713532784.40 total 117.15 last 176.72) - /unlink 20000 (time 1713532788.50 total 121.25 last 199.07) - /unlink 22119 (time 1713532798.50 total 131.25 last 211.85) - /unlink 24689 (time 1713532808.50 total 141.25 last 256.98) - /unlink 27858 (time 1713532818.51 total 151.25 last 316.85) - /unlink 30000 (time 1713532824.39 total 157.13 last 364.24) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 162.91 seconds: 199.87 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server unloading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2