== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:52:34 (1713520354) Stopping clients: oleg215-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg215-client.virtnet /mnt/lustre2 (opts:-f) oleg215-server: oleg215-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: detected 4 online CPUs by sysfs oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg215-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg215-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock oleg215-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2078 (time 1713520397.68 total 10.00 last 207.78) - open/close 4021 (time 1713520407.69 total 20.00 last 194.27) - open/close 6019 (time 1713520417.69 total 30.00 last 199.75) - open/close 7817 (time 1713520427.69 total 40.01 last 179.71) - open/close 10000 (time 1713520437.64 total 49.96 last 219.44) - open/close 13417 (time 1713520447.64 total 59.96 last 341.60) - open/close 16419 (time 1713520457.64 total 69.96 last 300.18) - open/close 18964 (time 1713520467.65 total 79.96 last 254.43) - open/close 19912 (time 1713520483.16 total 95.48 last 61.11) - open/close 21630 (time 1713520493.16 total 105.48 last 171.80) - open/close 23934 (time 1713520503.16 total 115.48 last 230.32) - open/close 26122 (time 1713520513.17 total 125.49 last 218.69) - open/close 28593 (time 1713520523.17 total 135.49 last 247.06) - open/close 30000 (time 1713520529.64 total 141.96 last 217.54) - open/close 32289 (time 1713520539.64 total 151.96 last 228.84) - open/close 34536 (time 1713520549.65 total 161.97 last 224.54) - open/close 36706 (time 1713520559.65 total 171.97 last 217.00) - open/close 40000 (time 1713520568.48 total 180.80 last 372.98) - open/close 43054 (time 1713520578.48 total 190.80 last 305.38) - open/close 45821 (time 1713520588.48 total 200.80 last 276.58) - open/close 48770 (time 1713520598.49 total 210.80 last 294.86) - open/close 50000 (time 1713520602.61 total 214.92 last 298.52) - open/close 53184 (time 1713520612.61 total 224.92 last 318.38) - open/close 57708 (time 1713520622.61 total 234.93 last 452.36) - open/close 60000 (time 1713520627.32 total 239.64 last 486.60) - open/close 63414 (time 1713520637.32 total 249.64 last 341.29) total: 64768 open/close in 253.19 seconds: 255.81 ops/second - /unlink 5773 (time 1713520651.74 total 10.00 last 577.27) - /unlink 10000 (time 1713520658.03 total 16.29 last 672.52) - /unlink 17667 (time 1713520668.03 total 26.29 last 766.62) - /unlink 20000 (time 1713520671.63 total 29.88 last 648.51) - /unlink 26166 (time 1713520681.63 total 39.89 last 616.48) - /unlink 30000 (time 1713520687.18 total 45.44 last 690.18) - /unlink 36837 (time 1713520697.18 total 55.44 last 683.69) - /unlink 40000 (time 1713520701.20 total 59.46 last 786.88) - /unlink 47024 (time 1713520711.20 total 69.46 last 702.35) - /unlink 50000 (time 1713520715.08 total 73.33 last 768.76) - /unlink 56712 (time 1713520725.08 total 83.33 last 671.09) - /unlink 60000 (time 1713520731.87 total 90.13 last 484.21) - /unlink 64152 (time 1713520741.87 total 100.13 last 415.16) total: 64768 /unlink in 101.83 seconds: 636.02 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2986 (time 1713520769.12 total 10.01 last 298.39) - open/close 6558 (time 1713520779.12 total 20.01 last 357.15) - open/close 8794 (time 1713520789.12 total 30.01 last 223.49) - open/close 10000 (time 1713520795.92 total 36.81 last 177.53) - open/close 12175 (time 1713520805.92 total 46.81 last 217.48) - open/close 14622 (time 1713520815.92 total 56.81 last 244.64) - open/close 17631 (time 1713520825.96 total 66.85 last 299.82) - open/close 20000 (time 1713520833.58 total 74.47 last 310.69) - open/close 22794 (time 1713520843.58 total 84.47 last 279.39) - open/close 25227 (time 1713520853.59 total 94.48 last 243.10) - open/close 28443 (time 1713520863.59 total 104.48 last 321.51) - open/close 30000 (time 1713520869.13 total 110.02 last 281.32) - open/close 32444 (time 1713520879.13 total 120.02 last 244.37) - open/close 34983 (time 1713520889.13 total 130.02 last 253.86) - open/close 36859 (time 1713520899.13 total 140.02 last 187.51) - open/close 39238 (time 1713520909.14 total 150.03 last 237.82) - open/close 40000 (time 1713520914.21 total 155.10 last 150.12) - open/close 41517 (time 1713520934.92 total 175.81 last 73.26) - open/close 43291 (time 1713520944.92 total 185.81 last 177.35) - open/close 45186 (time 1713520954.93 total 195.82 last 189.44) - open/close 47800 (time 1713520964.93 total 205.82 last 261.37) - open/close 50000 (time 1713520972.56 total 213.46 last 288.11) - open/close 52552 (time 1713520982.57 total 223.46 last 255.13) - open/close 54802 (time 1713520992.57 total 233.46 last 224.98) - open/close 58828 (time 1713521002.57 total 243.46 last 402.53) - open/close 60000 (time 1713521005.10 total 245.99 last 463.09) total: 64768 open/close in 255.67 seconds: 253.32 ops/second - /unlink 7979 (time 1713521025.58 total 10.00 last 797.87) - /unlink 10000 (time 1713521028.69 total 13.11 last 650.09) - /unlink 17155 (time 1713521038.69 total 23.11 last 715.40) - /unlink 20000 (time 1713521042.05 total 26.47 last 847.32) - /unlink 27866 (time 1713521052.05 total 36.47 last 786.60) - /unlink 30000 (time 1713521054.82 total 39.24 last 770.57) - /unlink 36639 (time 1713521064.82 total 49.24 last 663.85) - /unlink 40000 (time 1713521069.49 total 53.91 last 719.24) - /unlink 48034 (time 1713521079.49 total 63.91 last 803.40) - /unlink 50000 (time 1713521082.56 total 66.98 last 640.51) - /unlink 58249 (time 1713521092.56 total 76.98 last 824.87) - /unlink 60000 (time 1713521094.79 total 79.22 last 783.67) total: 64768 /unlink in 84.29 seconds: 768.36 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4877 (time 1713521129.58 total 10.00 last 487.64) - open/close 10000 (time 1713521138.82 total 19.24 last 554.40) - open/close 16025 (time 1713521148.82 total 29.24 last 602.48) - open/close 20000 (time 1713521156.28 total 36.70 last 532.95) - open/close 24276 (time 1713521166.28 total 46.70 last 427.60) - open/close 28138 (time 1713521176.28 total 56.70 last 386.15) - open/close 30000 (time 1713521181.17 total 61.59 last 381.29) - open/close 33511 (time 1713521191.17 total 71.59 last 351.04) - open/close 37206 (time 1713521201.17 total 81.59 last 369.45) - open/close 40000 (time 1713521207.26 total 87.68 last 458.44) - open/close 43908 (time 1713521217.26 total 97.68 last 390.78) - open/close 47773 (time 1713521227.27 total 107.68 last 386.45) - open/close 50000 (time 1713521232.75 total 113.17 last 406.36) - open/close 53916 (time 1713521242.75 total 123.17 last 391.53) - open/close 57507 (time 1713521252.75 total 133.17 last 359.06) - open/close 60000 (time 1713521259.90 total 140.32 last 348.51) - open/close 63579 (time 1713521269.90 total 150.32 last 357.82) total: 64768 open/close in 153.19 seconds: 422.80 ops/second - /unlink 6513 (time 1713521283.66 total 10.00 last 651.25) - /unlink 10000 (time 1713521288.37 total 14.71 last 740.60) - /unlink 18139 (time 1713521298.37 total 24.71 last 813.84) - /unlink 26356 (time 1713521308.37 total 34.71 last 821.68) - /unlink 30000 (time 1713521313.53 total 39.87 last 705.81) - /unlink 37257 (time 1713521323.53 total 49.87 last 725.65) - /unlink 40000 (time 1713521327.08 total 53.42 last 773.26) - /unlink 46207 (time 1713521337.08 total 63.42 last 620.69) - /unlink 50000 (time 1713521342.25 total 68.59 last 734.21) - /unlink 57897 (time 1713521352.25 total 78.59 last 789.62) - /unlink 60000 (time 1713521356.13 total 82.47 last 541.83) total: 64768 /unlink in 88.96 seconds: 728.05 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3137 (time 1713521395.37 total 10.00 last 313.65) - open/close 7542 (time 1713521405.37 total 20.00 last 440.38) - open/close 10000 (time 1713521411.13 total 25.76 last 426.79) - open/close 14034 (time 1713521421.13 total 35.77 last 403.20) - open/close 18320 (time 1713521431.13 total 45.77 last 428.53) - open/close 20000 (time 1713521434.96 total 49.59 last 439.48) - open/close 24685 (time 1713521444.97 total 59.60 last 468.05) - open/close 28632 (time 1713521454.97 total 69.60 last 394.61) - open/close 30000 (time 1713521458.52 total 73.16 last 384.88) - open/close 33786 (time 1713521468.53 total 83.16 last 378.52) - open/close 38076 (time 1713521478.53 total 93.16 last 428.97) - open/close 40000 (time 1713521482.72 total 97.35 last 459.09) - open/close 43966 (time 1713521492.72 total 107.35 last 396.58) - open/close 48200 (time 1713521502.72 total 117.35 last 423.40) - open/close 50000 (time 1713521507.72 total 122.35 last 360.14) - open/close 54462 (time 1713521517.72 total 132.35 last 446.17) - open/close 59688 (time 1713521527.72 total 142.35 last 522.55) total: 64768 open/close in 152.01 seconds: 426.08 ops/second - /unlink 8228 (time 1713521548.21 total 10.00 last 822.73) - /unlink 16931 (time 1713521558.21 total 20.00 last 870.28) - /unlink 20000 (time 1713521561.86 total 23.65 last 840.87) - /unlink 28188 (time 1713521571.86 total 33.65 last 818.67) - /unlink 30000 (time 1713521574.23 total 36.02 last 766.37) - /unlink 38238 (time 1713521584.23 total 46.02 last 823.73) - /unlink 46431 (time 1713521594.23 total 56.02 last 819.15) - /unlink 50000 (time 1713521598.24 total 60.03 last 889.65) - /unlink 57218 (time 1713521608.24 total 70.03 last 721.77) - /unlink 60000 (time 1713521612.39 total 74.18 last 671.26) total: 64768 /unlink in 80.17 seconds: 807.89 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4371 (time 1713521650.48 total 10.00 last 437.10) - open/close 9711 (time 1713521660.49 total 20.00 last 533.95) - open/close 15095 (time 1713521670.49 total 30.00 last 538.23) - open/close 19464 (time 1713521680.49 total 40.01 last 436.85) - open/close 24467 (time 1713521690.49 total 50.01 last 500.24) - open/close 29100 (time 1713521700.49 total 60.01 last 463.25) - open/close 34515 (time 1713521710.49 total 70.01 last 541.48) - open/close 39278 (time 1713521720.49 total 80.01 last 476.26) - open/close 44098 (time 1713521730.49 total 90.01 last 481.91) - open/close 49209 (time 1713521740.50 total 100.01 last 511.07) - open/close 54591 (time 1713521750.50 total 110.01 last 538.13) - open/close 59977 (time 1713521760.50 total 120.01 last 538.53) total: 64768 open/close in 129.50 seconds: 500.14 ops/second - /unlink 7883 (time 1713521780.84 total 10.00 last 788.25) - /unlink 10000 (time 1713521783.00 total 12.16 last 979.90) - /unlink 19631 (time 1713521793.00 total 22.16 last 963.10) - /unlink 28619 (time 1713521803.00 total 32.16 last 898.71) - /unlink 37012 (time 1713521813.00 total 42.16 last 839.24) - /unlink 40000 (time 1713521816.92 total 46.08 last 762.55) - /unlink 48611 (time 1713521826.92 total 56.08 last 861.00) - /unlink 57172 (time 1713521836.92 total 66.08 last 856.07) - /unlink 60000 (time 1713521840.20 total 69.36 last 863.74) total: 64768 /unlink in 74.22 seconds: 872.64 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server unloading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2