== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:41:21 (1713296481) Stopping clients: oleg249-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg249-client.virtnet /mnt/lustre2 (opts:-f) oleg249-server: oleg249-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: detected 4 online CPUs by sysfs oleg249-server: Force libcfs to create 2 CPU partitions oleg249-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg249-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg249-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock oleg249-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5330 (time 1713296514.66 total 10.00 last 532.97) - open/close 10000 (time 1713296523.56 total 18.90 last 524.51) - open/close 15257 (time 1713296533.56 total 28.90 last 525.70) - open/close 20000 (time 1713296543.01 total 38.35 last 502.23) - open/close 24976 (time 1713296553.01 total 48.35 last 497.54) - open/close 30000 (time 1713296562.98 total 58.32 last 503.86) - open/close 34742 (time 1713296572.98 total 68.32 last 474.20) - open/close 40000 (time 1713296582.93 total 78.27 last 528.35) - open/close 45192 (time 1713296592.93 total 88.27 last 519.10) - open/close 50000 (time 1713296601.82 total 97.17 last 540.73) - open/close 55058 (time 1713296611.83 total 107.17 last 505.76) - open/close 60000 (time 1713296620.67 total 116.01 last 558.67) total: 64768 open/close in 125.30 seconds: 516.92 ops/second - /unlink 8561 (time 1713296640.84 total 10.00 last 856.04) - /unlink 16770 (time 1713296650.84 total 20.00 last 820.81) - /unlink 20000 (time 1713296654.81 total 23.98 last 812.45) - /unlink 28178 (time 1713296664.81 total 33.98 last 817.73) - /unlink 30000 (time 1713296666.90 total 36.07 last 872.67) - /unlink 38552 (time 1713296676.90 total 46.07 last 855.14) - /unlink 47127 (time 1713296686.90 total 56.07 last 857.46) - /unlink 50000 (time 1713296690.45 total 59.61 last 810.86) - /unlink 58076 (time 1713296700.45 total 69.61 last 807.48) - /unlink 60000 (time 1713296702.83 total 71.99 last 808.24) total: 64768 /unlink in 77.56 seconds: 835.05 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5092 (time 1713296738.81 total 10.00 last 509.16) - open/close 10000 (time 1713296748.76 total 19.95 last 493.09) - open/close 14787 (time 1713296758.76 total 29.96 last 478.62) - open/close 19762 (time 1713296768.76 total 39.96 last 497.44) - open/close 24724 (time 1713296778.76 total 49.96 last 496.19) - open/close 29876 (time 1713296788.76 total 59.96 last 515.12) - open/close 35061 (time 1713296798.77 total 69.96 last 518.40) - open/close 40000 (time 1713296808.21 total 79.41 last 522.71) - open/close 45029 (time 1713296818.21 total 89.41 last 502.90) - open/close 50000 (time 1713296828.19 total 99.39 last 498.09) - open/close 54942 (time 1713296838.19 total 109.39 last 494.18) - open/close 59880 (time 1713296848.20 total 119.39 last 493.77) total: 64768 open/close in 128.68 seconds: 503.31 ops/second - /unlink 9264 (time 1713296868.30 total 10.00 last 926.39) - /unlink 18347 (time 1713296878.30 total 20.00 last 908.29) - /unlink 27523 (time 1713296888.30 total 30.00 last 917.54) - /unlink 30000 (time 1713296891.06 total 32.76 last 896.14) - /unlink 40000 (time 1713296901.03 total 42.74 last 1002.73) - /unlink 49885 (time 1713296911.03 total 52.74 last 988.49) - /unlink 59780 (time 1713296921.03 total 62.74 last 989.41) total: 64768 /unlink in 68.19 seconds: 949.77 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4727 (time 1713296968.57 total 10.00 last 472.67) - open/close 9627 (time 1713296978.58 total 20.00 last 489.92) - open/close 14565 (time 1713296988.58 total 30.00 last 493.75) - open/close 19536 (time 1713296998.58 total 40.01 last 497.00) - open/close 24565 (time 1713297008.58 total 50.01 last 502.82) - open/close 29351 (time 1713297018.58 total 60.01 last 478.57) - open/close 34406 (time 1713297028.58 total 70.01 last 505.47) - open/close 39394 (time 1713297038.58 total 80.01 last 498.73) - open/close 44008 (time 1713297048.58 total 90.01 last 461.39) - open/close 48682 (time 1713297058.59 total 100.02 last 467.04) - open/close 50000 (time 1713297061.27 total 102.69 last 492.37) - open/close 54885 (time 1713297071.27 total 112.70 last 488.45) - open/close 60000 (time 1713297081.21 total 122.64 last 514.47) total: 64768 open/close in 132.00 seconds: 490.65 ops/second - /unlink 8565 (time 1713297101.44 total 10.00 last 856.48) - /unlink 17217 (time 1713297111.44 total 20.00 last 865.20) - /unlink 20000 (time 1713297114.68 total 23.24 last 859.65) - /unlink 28526 (time 1713297124.68 total 33.24 last 852.58) - /unlink 37169 (time 1713297134.68 total 43.24 last 864.29) - /unlink 40000 (time 1713297137.93 total 46.49 last 870.72) - /unlink 48800 (time 1713297147.93 total 56.49 last 879.94) - /unlink 57599 (time 1713297157.93 total 66.49 last 879.83) - /unlink 60000 (time 1713297160.64 total 69.20 last 885.89) total: 64768 /unlink in 75.04 seconds: 863.07 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4829 (time 1713297198.66 total 10.00 last 482.82) - open/close 9701 (time 1713297208.66 total 20.00 last 487.14) - open/close 14670 (time 1713297218.66 total 30.00 last 496.85) - open/close 19649 (time 1713297228.66 total 40.01 last 497.82) - open/close 24701 (time 1713297238.66 total 50.01 last 505.13) - open/close 28869 (time 1713297248.66 total 60.01 last 416.77) - open/close 30000 (time 1713297251.93 total 63.27 last 346.41) - open/close 33042 (time 1713297261.93 total 73.27 last 304.18) - open/close 37183 (time 1713297271.93 total 83.27 last 414.09) - open/close 40000 (time 1713297277.59 total 88.94 last 497.30) - open/close 43178 (time 1713297287.59 total 98.94 last 317.77) - open/close 46983 (time 1713297297.59 total 108.94 last 380.45) - open/close 50000 (time 1713297306.92 total 118.26 last 323.67) - open/close 53105 (time 1713297316.92 total 128.26 last 310.46) - open/close 57256 (time 1713297326.92 total 138.26 last 415.03) - open/close 60000 (time 1713297334.81 total 146.15 last 347.76) - open/close 62763 (time 1713297344.81 total 156.16 last 276.27) total: 64768 open/close in 162.68 seconds: 398.13 ops/second - /unlink 6099 (time 1713297362.39 total 10.00 last 609.88) - /unlink 10000 (time 1713297369.23 total 16.84 last 570.26) - /unlink 15463 (time 1713297379.23 total 26.84 last 546.24) - /unlink 20000 (time 1713297388.10 total 35.71 last 511.63) - /unlink 25778 (time 1713297398.10 total 45.71 last 577.74) - /unlink 30000 (time 1713297405.27 total 52.88 last 588.91) - /unlink 35777 (time 1713297415.27 total 62.88 last 577.67) - /unlink 40000 (time 1713297423.33 total 70.94 last 523.82) - /unlink 44422 (time 1713297433.33 total 80.94 last 442.12) - /unlink 49203 (time 1713297443.33 total 90.94 last 478.09) - /unlink 54037 (time 1713297453.33 total 100.95 last 483.37) - /unlink 58644 (time 1713297463.33 total 110.95 last 460.65) - /unlink 60000 (time 1713297466.51 total 114.12 last 427.04) - /unlink 64569 (time 1713297476.51 total 124.12 last 456.87) total: 64768 /unlink in 124.57 seconds: 519.92 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3252 (time 1713297505.49 total 10.00 last 325.16) - open/close 6472 (time 1713297515.49 total 20.00 last 321.96) - open/close 9162 (time 1713297525.49 total 30.01 last 268.93) - open/close 10000 (time 1713297528.70 total 33.21 last 261.67) - open/close 12758 (time 1713297538.70 total 43.21 last 275.69) - open/close 15578 (time 1713297548.70 total 53.21 last 281.91) - open/close 18062 (time 1713297558.70 total 63.22 last 248.39) - open/close 20000 (time 1713297565.17 total 69.68 last 299.95) - open/close 22673 (time 1713297575.17 total 79.68 last 267.25) - open/close 25181 (time 1713297585.17 total 89.68 last 250.72) - open/close 27886 (time 1713297595.17 total 99.68 last 270.43) - open/close 30000 (time 1713297603.45 total 107.96 last 255.52) - open/close 32663 (time 1713297613.45 total 117.96 last 266.25) - open/close 35814 (time 1713297623.45 total 127.96 last 315.05) - open/close 38901 (time 1713297633.45 total 137.96 last 308.65) - open/close 40000 (time 1713297637.77 total 142.28 last 254.26) - open/close 42540 (time 1713297647.77 total 152.29 last 253.96) - open/close 44994 (time 1713297657.78 total 162.29 last 245.36) - open/close 47588 (time 1713297667.78 total 172.29 last 259.35) - open/close 50000 (time 1713297674.02 total 178.53 last 386.65) - open/close 52818 (time 1713297684.02 total 188.53 last 281.71) - open/close 55880 (time 1713297694.02 total 198.53 last 306.20) - open/close 60000 (time 1713297703.55 total 208.06 last 432.25) - open/close 63548 (time 1713297713.55 total 218.06 last 354.78) total: 64768 open/close in 222.12 seconds: 291.59 ops/second - /unlink 5092 (time 1713297728.57 total 10.00 last 509.16) - /unlink 9777 (time 1713297738.57 total 20.00 last 468.45) - /unlink 14318 (time 1713297748.57 total 30.00 last 454.02) - /unlink 18844 (time 1713297758.57 total 40.00 last 452.54) - /unlink 20000 (time 1713297761.29 total 42.72 last 425.48) - /unlink 24429 (time 1713297771.29 total 52.72 last 442.87) - /unlink 28993 (time 1713297781.29 total 62.72 last 456.30) - /unlink 33600 (time 1713297791.29 total 72.73 last 460.63) - /unlink 38293 (time 1713297801.30 total 82.73 last 469.26) - /unlink 40000 (time 1713297803.60 total 85.03 last 741.10) - /unlink 45108 (time 1713297813.60 total 95.03 last 510.67) - /unlink 49928 (time 1713297823.60 total 105.04 last 481.89) - /unlink 54827 (time 1713297833.60 total 115.04 last 489.88) - /unlink 60000 (time 1713297840.15 total 121.58 last 790.28) total: 64768 /unlink in 127.87 seconds: 506.53 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server unloading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2