== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:41:16 (1713429676) Stopping clients: oleg105-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg105-client.virtnet /mnt/lustre2 (opts:-f) oleg105-server: oleg105-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: detected 4 online CPUs by sysfs oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg105-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg105-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock oleg105-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2888 (time 1713429718.80 total 10.00 last 288.72) - open/close 5574 (time 1713429728.80 total 20.01 last 268.53) - open/close 8434 (time 1713429738.81 total 30.01 last 285.97) - open/close 10000 (time 1713429744.39 total 35.59 last 280.50) - open/close 12202 (time 1713429754.39 total 45.59 last 220.09) - open/close 14702 (time 1713429764.40 total 55.60 last 249.91) - open/close 16908 (time 1713429774.40 total 65.60 last 220.56) - open/close 18777 (time 1713429784.40 total 75.60 last 186.89) - open/close 20000 (time 1713429790.36 total 81.56 last 205.25) - open/close 22182 (time 1713429800.36 total 91.56 last 218.19) - open/close 24448 (time 1713429810.36 total 101.56 last 226.53) - open/close 26479 (time 1713429820.37 total 111.57 last 203.04) - open/close 28470 (time 1713429830.37 total 121.57 last 199.02) - open/close 30000 (time 1713429838.33 total 129.53 last 192.17) - open/close 30398 (time 1713429851.34 total 142.54 last 30.60) - open/close 32817 (time 1713429861.34 total 152.54 last 241.87) - open/close 34911 (time 1713429871.34 total 162.54 last 209.38) - open/close 37117 (time 1713429881.34 total 172.54 last 220.56) - open/close 39360 (time 1713429891.34 total 182.54 last 224.28) - open/close 40000 (time 1713429894.30 total 185.50 last 216.53) - open/close 42079 (time 1713429904.30 total 195.50 last 207.86) - open/close 44201 (time 1713429914.30 total 205.50 last 212.14) - open/close 46310 (time 1713429924.30 total 215.50 last 210.87) - open/close 48754 (time 1713429934.31 total 225.51 last 244.33) - open/close 50000 (time 1713429938.66 total 229.86 last 285.94) - open/close 52918 (time 1713429948.66 total 239.87 last 291.80) - open/close 55836 (time 1713429958.67 total 249.87 last 291.78) - open/close 59188 (time 1713429968.67 total 259.87 last 335.18) - open/close 60000 (time 1713429971.35 total 262.55 last 302.57) - open/close 63473 (time 1713429981.35 total 272.55 last 347.24) total: 64768 open/close in 276.58 seconds: 234.18 ops/second - /unlink 6727 (time 1713429996.29 total 10.00 last 672.66) - /unlink 10000 (time 1713430000.13 total 13.84 last 852.85) - /unlink 19205 (time 1713430010.13 total 23.84 last 920.40) - /unlink 27480 (time 1713430020.13 total 33.84 last 827.46) - /unlink 30000 (time 1713430022.85 total 36.56 last 928.00) - /unlink 38076 (time 1713430032.85 total 46.56 last 807.55) - /unlink 40000 (time 1713430035.29 total 49.00 last 786.78) - /unlink 48103 (time 1713430045.29 total 59.00 last 810.29) - /unlink 50000 (time 1713430047.41 total 61.12 last 896.07) - /unlink 58405 (time 1713430057.41 total 71.12 last 840.45) total: 64768 /unlink in 78.67 seconds: 823.26 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4693 (time 1713430097.48 total 10.00 last 469.29) - open/close 9015 (time 1713430107.48 total 20.00 last 432.15) - open/close 10000 (time 1713430109.68 total 22.20 last 447.51) - open/close 14720 (time 1713430119.68 total 32.20 last 471.94) - open/close 19246 (time 1713430129.68 total 42.20 last 452.55) - open/close 24587 (time 1713430139.68 total 52.21 last 534.03) - open/close 29730 (time 1713430149.68 total 62.21 last 514.29) - open/close 35009 (time 1713430159.68 total 72.21 last 527.89) - open/close 40000 (time 1713430169.53 total 82.05 last 506.96) - open/close 45263 (time 1713430179.53 total 92.05 last 526.26) - open/close 50000 (time 1713430188.33 total 100.85 last 538.47) - open/close 55586 (time 1713430198.33 total 110.85 last 558.58) - open/close 60000 (time 1713430206.02 total 118.55 last 573.47) total: 64768 open/close in 127.01 seconds: 509.95 ops/second - /unlink 8884 (time 1713430225.28 total 10.00 last 888.23) - /unlink 17508 (time 1713430235.28 total 20.00 last 862.35) - /unlink 20000 (time 1713430238.05 total 22.77 last 900.56) - /unlink 29405 (time 1713430248.05 total 32.77 last 940.45) - /unlink 37038 (time 1713430258.05 total 42.77 last 763.24) - /unlink 40000 (time 1713430261.51 total 46.23 last 856.25) - /unlink 49295 (time 1713430271.51 total 56.23 last 929.43) - /unlink 59319 (time 1713430281.51 total 66.23 last 1002.36) total: 64768 /unlink in 72.07 seconds: 898.72 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5192 (time 1713430322.99 total 10.00 last 519.15) - open/close 9910 (time 1713430332.99 total 20.00 last 471.76) - open/close 15483 (time 1713430342.99 total 30.00 last 557.25) - open/close 20000 (time 1713430350.88 total 37.89 last 572.43) - open/close 25463 (time 1713430360.88 total 47.89 last 546.23) - open/close 30000 (time 1713430368.84 total 55.85 last 570.09) - open/close 35987 (time 1713430378.84 total 65.85 last 598.64) - open/close 40000 (time 1713430385.76 total 72.78 last 579.74) - open/close 45671 (time 1713430395.76 total 82.78 last 567.04) - open/close 50000 (time 1713430404.33 total 91.34 last 505.44) - open/close 55318 (time 1713430414.33 total 101.34 last 531.72) - open/close 60000 (time 1713430423.02 total 110.03 last 538.73) total: 64768 open/close in 118.35 seconds: 547.27 ops/second - /unlink 9432 (time 1713430442.13 total 10.00 last 943.18) - /unlink 19410 (time 1713430452.13 total 20.00 last 997.74) - /unlink 27701 (time 1713430462.13 total 30.00 last 829.05) - /unlink 30000 (time 1713430464.89 total 32.76 last 833.67) - /unlink 37875 (time 1713430474.89 total 42.76 last 787.48) - /unlink 40000 (time 1713430477.58 total 45.45 last 789.63) - /unlink 48381 (time 1713430487.58 total 55.45 last 838.01) - /unlink 57080 (time 1713430497.58 total 65.45 last 869.84) - /unlink 60000 (time 1713430501.29 total 69.16 last 787.45) total: 64768 /unlink in 74.65 seconds: 867.58 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5421 (time 1713430542.94 total 10.00 last 542.05) - open/close 10000 (time 1713430552.24 total 19.31 last 492.04) - open/close 14918 (time 1713430562.24 total 29.31 last 491.78) - open/close 20000 (time 1713430572.15 total 39.21 last 513.04) - open/close 24819 (time 1713430582.15 total 49.21 last 481.85) - open/close 29824 (time 1713430592.15 total 59.22 last 500.45) - open/close 35103 (time 1713430602.15 total 69.22 last 527.84) - open/close 40000 (time 1713430612.03 total 79.09 last 495.88) - open/close 44995 (time 1713430622.03 total 89.09 last 499.43) - open/close 50000 (time 1713430631.64 total 98.70 last 520.92) - open/close 54891 (time 1713430641.64 total 108.70 last 489.05) - open/close 59634 (time 1713430651.64 total 118.70 last 474.20) - open/close 64464 (time 1713430661.64 total 128.71 last 482.95) total: 64768 open/close in 129.34 seconds: 500.77 ops/second - /unlink 8387 (time 1713430673.10 total 10.00 last 838.66) - /unlink 16675 (time 1713430683.10 total 20.00 last 828.72) - /unlink 20000 (time 1713430687.26 total 24.16 last 799.21) - /unlink 28116 (time 1713430697.26 total 34.16 last 811.50) - /unlink 30000 (time 1713430699.60 total 36.50 last 805.85) - /unlink 38382 (time 1713430709.60 total 46.50 last 838.18) - /unlink 46845 (time 1713430719.60 total 56.50 last 846.23) - /unlink 50000 (time 1713430723.33 total 60.23 last 846.89) - /unlink 58730 (time 1713430733.33 total 70.23 last 872.92) total: 64768 /unlink in 76.94 seconds: 841.79 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5357 (time 1713430773.41 total 10.00 last 535.60) - open/close 10000 (time 1713430782.28 total 18.88 last 523.17) - open/close 15378 (time 1713430792.28 total 28.88 last 537.72) - open/close 20000 (time 1713430800.92 total 37.52 last 534.98) - open/close 25213 (time 1713430810.92 total 47.52 last 521.29) - open/close 30000 (time 1713430819.96 total 56.56 last 529.55) - open/close 35379 (time 1713430829.96 total 66.56 last 537.86) - open/close 40000 (time 1713430838.78 total 75.37 last 524.19) - open/close 45133 (time 1713430848.78 total 85.38 last 513.23) - open/close 50000 (time 1713430857.90 total 94.49 last 533.75) - open/close 55106 (time 1713430867.90 total 104.49 last 510.57) - open/close 60000 (time 1713430877.21 total 113.80 last 525.84) total: 64768 open/close in 122.79 seconds: 527.48 ops/second - /unlink 8594 (time 1713430897.02 total 10.00 last 859.38) - /unlink 17174 (time 1713430907.02 total 20.00 last 857.92) - /unlink 20000 (time 1713430910.31 total 23.29 last 858.89) - /unlink 28437 (time 1713430920.32 total 33.29 last 843.60) - /unlink 37110 (time 1713430930.32 total 43.29 last 867.30) - /unlink 40000 (time 1713430933.58 total 46.56 last 885.11) - /unlink 48436 (time 1713430943.58 total 56.56 last 843.55) - /unlink 57357 (time 1713430953.58 total 66.56 last 892.01) - /unlink 60000 (time 1713430956.46 total 69.44 last 917.08) total: 64768 /unlink in 74.68 seconds: 867.30 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server unloading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2