== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:24:08 (1713425048) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock oleg408-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5790 (time 1713425080.42 total 10.00 last 578.96) - open/close 10000 (time 1713425087.85 total 17.42 last 567.23) - open/close 15635 (time 1713425097.85 total 27.42 last 563.48) - open/close 20000 (time 1713425105.54 total 35.11 last 567.63) - open/close 25622 (time 1713425115.54 total 45.11 last 562.15) - open/close 30000 (time 1713425123.37 total 52.95 last 558.70) - open/close 35595 (time 1713425133.38 total 62.95 last 559.40) - open/close 40000 (time 1713425141.13 total 70.71 last 567.98) - open/close 45493 (time 1713425151.19 total 80.77 last 545.95) - open/close 50000 (time 1713425159.63 total 89.21 last 533.99) - open/close 55066 (time 1713425169.63 total 99.21 last 506.56) - open/close 60000 (time 1713425178.96 total 108.54 last 529.02) total: 64768 open/close in 118.03 seconds: 548.76 ops/second - /unlink 8562 (time 1713425199.26 total 10.00 last 856.13) - /unlink 17394 (time 1713425209.26 total 20.00 last 883.13) - /unlink 20000 (time 1713425212.70 total 23.44 last 757.53) - /unlink 28810 (time 1713425222.70 total 33.44 last 880.97) - /unlink 37092 (time 1713425232.70 total 43.44 last 828.14) - /unlink 40000 (time 1713425236.80 total 47.54 last 709.22) - /unlink 47439 (time 1713425246.80 total 57.54 last 743.87) - /unlink 50000 (time 1713425249.71 total 60.46 last 879.32) - /unlink 57046 (time 1713425259.71 total 70.46 last 704.60) - /unlink 60000 (time 1713425263.08 total 73.82 last 878.26) total: 64768 /unlink in 79.64 seconds: 813.29 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5057 (time 1713425299.36 total 10.00 last 505.65) - open/close 10000 (time 1713425309.02 total 19.66 last 511.70) - open/close 14968 (time 1713425319.02 total 29.66 last 496.69) - open/close 20000 (time 1713425328.65 total 39.29 last 522.58) - open/close 25276 (time 1713425338.65 total 49.29 last 527.57) - open/close 30000 (time 1713425347.60 total 58.25 last 527.60) - open/close 35359 (time 1713425357.61 total 68.25 last 535.82) - open/close 40000 (time 1713425366.20 total 76.84 last 540.05) - open/close 45136 (time 1713425376.20 total 86.84 last 513.49) - open/close 49745 (time 1713425386.20 total 96.85 last 460.82) - open/close 54407 (time 1713425396.20 total 106.85 last 466.20) - open/close 59546 (time 1713425406.20 total 116.85 last 513.84) - open/close 64476 (time 1713425416.21 total 126.85 last 492.94) total: 64768 open/close in 127.44 seconds: 508.24 ops/second - /unlink 8961 (time 1713425427.62 total 10.00 last 896.09) - /unlink 17824 (time 1713425437.62 total 20.00 last 886.22) - /unlink 20000 (time 1713425440.12 total 22.50 last 870.87) - /unlink 28679 (time 1713425450.12 total 32.50 last 867.86) - /unlink 37686 (time 1713425460.12 total 42.50 last 900.60) - /unlink 40000 (time 1713425462.62 total 45.00 last 924.87) - /unlink 48677 (time 1713425472.62 total 55.00 last 867.63) - /unlink 57648 (time 1713425482.62 total 65.00 last 897.07) - /unlink 60000 (time 1713425485.20 total 67.58 last 913.21) total: 64768 /unlink in 72.87 seconds: 888.80 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5094 (time 1713425525.22 total 10.00 last 509.36) - open/close 9337 (time 1713425535.22 total 20.00 last 424.29) - open/close 14163 (time 1713425545.22 total 30.00 last 482.54) - open/close 18396 (time 1713425555.22 total 40.00 last 423.24) - open/close 20000 (time 1713425558.11 total 42.90 last 554.20) - open/close 25257 (time 1713425568.12 total 52.90 last 525.61) - open/close 29620 (time 1713425578.12 total 62.90 last 436.24) - open/close 33894 (time 1713425588.12 total 72.90 last 427.40) - open/close 38124 (time 1713425598.12 total 82.90 last 422.99) - open/close 40000 (time 1713425603.83 total 88.61 last 328.52) - open/close 43844 (time 1713425613.83 total 98.61 last 384.39) - open/close 48433 (time 1713425623.83 total 108.61 last 458.86) - open/close 50000 (time 1713425626.92 total 111.71 last 506.43) - open/close 54774 (time 1713425636.92 total 121.71 last 477.36) - open/close 58902 (time 1713425646.93 total 131.71 last 412.73) - open/close 60000 (time 1713425649.10 total 133.88 last 505.71) total: 64768 open/close in 143.00 seconds: 452.92 ops/second - /unlink 8837 (time 1713425669.04 total 10.00 last 883.61) - /unlink 17701 (time 1713425679.04 total 20.00 last 886.32) - /unlink 20000 (time 1713425681.58 total 22.54 last 905.11) - /unlink 28885 (time 1713425691.58 total 32.54 last 888.49) - /unlink 37894 (time 1713425701.58 total 42.54 last 900.86) - /unlink 40000 (time 1713425703.92 total 44.89 last 898.93) - /unlink 49134 (time 1713425713.92 total 54.89 last 913.32) - /unlink 57447 (time 1713425723.92 total 64.89 last 831.28) - /unlink 60000 (time 1713425726.71 total 67.67 last 916.82) total: 64768 /unlink in 72.74 seconds: 890.42 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5514 (time 1713425769.83 total 10.00 last 551.38) - open/close 10000 (time 1713425778.33 total 18.50 last 527.80) - open/close 15177 (time 1713425788.33 total 28.50 last 517.63) - open/close 20000 (time 1713425797.87 total 38.04 last 505.51) - open/close 24531 (time 1713425807.88 total 48.04 last 453.08) - open/close 28233 (time 1713425817.88 total 58.04 last 370.14) - open/close 30000 (time 1713425820.90 total 61.07 last 584.45) - open/close 35681 (time 1713425830.90 total 71.07 last 568.09) - open/close 40000 (time 1713425839.85 total 80.02 last 482.65) - open/close 44925 (time 1713425849.85 total 90.02 last 492.48) - open/close 50000 (time 1713425858.40 total 98.57 last 593.39) - open/close 55106 (time 1713425868.40 total 108.57 last 510.56) - open/close 60000 (time 1713425877.52 total 117.69 last 536.49) total: 64768 open/close in 126.69 seconds: 511.24 ops/second - /unlink 8324 (time 1713425897.33 total 10.00 last 832.31) - /unlink 10000 (time 1713425899.44 total 12.11 last 794.30) - /unlink 18282 (time 1713425909.44 total 22.11 last 828.19) - /unlink 26841 (time 1713425919.44 total 32.11 last 855.86) - /unlink 30000 (time 1713425923.08 total 35.76 last 866.75) - /unlink 38534 (time 1713425933.08 total 45.76 last 853.39) - /unlink 46741 (time 1713425943.08 total 55.76 last 820.63) - /unlink 50000 (time 1713425947.43 total 60.10 last 750.08) - /unlink 58316 (time 1713425957.43 total 70.10 last 831.56) - /unlink 60000 (time 1713425959.50 total 72.17 last 814.01) total: 64768 /unlink in 77.36 seconds: 837.18 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3161 (time 1713425996.32 total 10.00 last 316.02) - open/close 5860 (time 1713426006.32 total 20.00 last 269.90) - open/close 8794 (time 1713426016.32 total 30.00 last 293.39) - open/close 10000 (time 1713426019.43 total 33.12 last 387.02) - open/close 12907 (time 1713426029.43 total 43.12 last 290.67) - open/close 15847 (time 1713426039.44 total 53.12 last 293.91) - open/close 18566 (time 1713426049.44 total 63.13 last 271.84) - open/close 20000 (time 1713426054.70 total 68.39 last 272.48) - open/close 23690 (time 1713426064.70 total 78.39 last 368.97) - open/close 27266 (time 1713426074.70 total 88.39 last 357.59) - open/close 30000 (time 1713426082.21 total 95.90 last 364.19) - open/close 34157 (time 1713426092.21 total 105.90 last 415.63) - open/close 37260 (time 1713426102.21 total 115.90 last 310.28) - open/close 40000 (time 1713426107.00 total 120.69 last 572.13) - open/close 43652 (time 1713426117.00 total 130.69 last 365.19) - open/close 47093 (time 1713426127.00 total 140.69 last 343.98) - open/close 50000 (time 1713426136.58 total 150.26 last 303.70) - open/close 52480 (time 1713426146.58 total 160.26 last 247.99) - open/close 55197 (time 1713426156.58 total 170.27 last 271.66) - open/close 58368 (time 1713426166.58 total 180.27 last 317.00) - open/close 60000 (time 1713426172.28 total 185.97 last 286.15) - open/close 63352 (time 1713426182.29 total 195.97 last 335.15) total: 64768 open/close in 201.00 seconds: 322.22 ops/second - /unlink 4823 (time 1713426198.56 total 10.00 last 482.22) - /unlink 10000 (time 1713426207.82 total 19.26 last 559.01) - /unlink 15686 (time 1713426217.82 total 29.26 last 568.56) - /unlink 20000 (time 1713426227.35 total 38.79 last 452.95) - /unlink 24478 (time 1713426237.35 total 48.79 last 447.71) - /unlink 29030 (time 1713426247.35 total 58.79 last 455.18) - /unlink 30000 (time 1713426249.42 total 60.86 last 468.17) - /unlink 34770 (time 1713426259.42 total 70.86 last 476.99) - /unlink 39628 (time 1713426269.42 total 80.86 last 485.77) - /unlink 44231 (time 1713426279.42 total 90.86 last 460.27) - /unlink 49090 (time 1713426289.42 total 100.86 last 485.85) - /unlink 54266 (time 1713426299.43 total 110.87 last 517.55) - /unlink 59562 (time 1713426309.43 total 120.87 last 529.57) - /unlink 64605 (time 1713426319.43 total 130.87 last 504.28) total: 64768 /unlink in 131.12 seconds: 493.96 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server unloading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2