== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:50:28 (1713279028) Stopping clients: oleg153-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg153-client.virtnet /mnt/lustre2 (opts:-f) oleg153-server: oleg153-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: detected 4 online CPUs by sysfs oleg153-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg153-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg153-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock oleg153-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5288 (time 1713279062.48 total 10.00 last 528.74) - open/close 10000 (time 1713279071.32 total 18.83 last 533.41) - open/close 15843 (time 1713279081.32 total 28.84 last 584.25) - open/close 20000 (time 1713279088.61 total 36.13 last 569.74) - open/close 26113 (time 1713279098.61 total 46.13 last 611.27) - open/close 30000 (time 1713279105.21 total 52.73 last 588.87) - open/close 35956 (time 1713279115.21 total 62.73 last 595.57) - open/close 40000 (time 1713279122.72 total 70.24 last 538.48) - open/close 45377 (time 1713279132.72 total 80.24 last 537.68) - open/close 50000 (time 1713279141.48 total 89.00 last 527.95) - open/close 55617 (time 1713279151.48 total 99.00 last 561.62) - open/close 60000 (time 1713279159.21 total 106.73 last 567.48) total: 64768 open/close in 115.97 seconds: 558.49 ops/second - /unlink 8680 (time 1713279179.37 total 10.00 last 868.00) - /unlink 17288 (time 1713279189.37 total 20.00 last 860.77) - /unlink 20000 (time 1713279192.53 total 23.16 last 858.44) - /unlink 28195 (time 1713279202.53 total 33.16 last 819.49) - /unlink 30000 (time 1713279204.79 total 35.42 last 800.07) - /unlink 38427 (time 1713279214.79 total 45.42 last 842.64) - /unlink 46745 (time 1713279224.79 total 55.42 last 831.79) - /unlink 50000 (time 1713279228.80 total 59.43 last 811.80) - /unlink 58536 (time 1713279238.80 total 69.43 last 853.59) total: 64768 /unlink in 77.08 seconds: 840.25 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4969 (time 1713279276.06 total 10.00 last 496.85) - open/close 9633 (time 1713279286.10 total 20.04 last 464.38) - open/close 14746 (time 1713279296.10 total 30.05 last 511.20) - open/close 19785 (time 1713279306.10 total 40.05 last 503.88) - open/close 24454 (time 1713279316.10 total 50.05 last 466.87) - open/close 29353 (time 1713279326.10 total 60.05 last 489.90) - open/close 34322 (time 1713279336.10 total 70.05 last 496.88) - open/close 39310 (time 1713279346.10 total 80.05 last 498.77) - open/close 44064 (time 1713279356.11 total 90.05 last 475.36) - open/close 49153 (time 1713279366.11 total 100.05 last 508.82) - open/close 54476 (time 1713279376.11 total 110.05 last 532.21) - open/close 59746 (time 1713279386.11 total 120.05 last 526.94) total: 64768 open/close in 129.94 seconds: 498.44 ops/second - /unlink 8527 (time 1713279406.88 total 10.00 last 852.66) - /unlink 17217 (time 1713279416.88 total 20.00 last 868.98) - /unlink 20000 (time 1713279420.03 total 23.15 last 883.74) - /unlink 28731 (time 1713279430.03 total 33.15 last 873.04) - /unlink 37491 (time 1713279440.03 total 43.15 last 875.97) - /unlink 40000 (time 1713279442.92 total 46.03 last 870.34) - /unlink 48788 (time 1713279452.92 total 56.03 last 878.73) - /unlink 57621 (time 1713279462.92 total 66.03 last 883.29) - /unlink 60000 (time 1713279465.59 total 68.71 last 890.84) total: 64768 /unlink in 73.92 seconds: 876.19 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5170 (time 1713279506.74 total 10.00 last 516.94) - open/close 10000 (time 1713279515.54 total 18.80 last 548.99) - open/close 15405 (time 1713279525.54 total 28.80 last 540.46) - open/close 20000 (time 1713279534.03 total 37.29 last 541.50) - open/close 25282 (time 1713279544.03 total 47.29 last 528.18) - open/close 30000 (time 1713279552.80 total 56.05 last 538.11) - open/close 34677 (time 1713279562.80 total 66.06 last 467.63) - open/close 38726 (time 1713279572.80 total 76.06 last 404.87) - open/close 40000 (time 1713279575.39 total 78.64 last 492.13) - open/close 44498 (time 1713279585.39 total 88.65 last 449.78) - open/close 48846 (time 1713279595.39 total 98.65 last 434.76) - open/close 50000 (time 1713279597.56 total 100.82 last 531.69) - open/close 54471 (time 1713279607.56 total 110.82 last 447.04) - open/close 58713 (time 1713279617.56 total 120.82 last 424.12) - open/close 60000 (time 1713279620.17 total 123.43 last 493.85) total: 64768 open/close in 132.38 seconds: 489.27 ops/second - /unlink 7620 (time 1713279639.95 total 10.00 last 761.92) - /unlink 10000 (time 1713279642.58 total 12.63 last 905.39) - /unlink 17944 (time 1713279652.58 total 22.63 last 794.33) - /unlink 20000 (time 1713279654.88 total 24.93 last 893.14) - /unlink 28336 (time 1713279664.88 total 34.93 last 833.52) - /unlink 36513 (time 1713279674.88 total 44.93 last 817.68) - /unlink 40000 (time 1713279679.10 total 49.15 last 827.54) - /unlink 48241 (time 1713279689.10 total 59.15 last 824.04) - /unlink 50000 (time 1713279691.67 total 61.72 last 684.44) - /unlink 56098 (time 1713279701.67 total 71.72 last 609.77) - /unlink 60000 (time 1713279707.57 total 77.63 last 660.57) total: 64768 /unlink in 84.92 seconds: 762.65 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4281 (time 1713279741.40 total 10.00 last 428.03) - open/close 8182 (time 1713279751.40 total 20.00 last 390.07) - open/close 10000 (time 1713279755.93 total 24.53 last 401.15) - open/close 12835 (time 1713279765.93 total 34.54 last 283.45) - open/close 15708 (time 1713279775.93 total 44.54 last 287.29) - open/close 20000 (time 1713279785.38 total 53.98 last 454.56) - open/close 24847 (time 1713279795.38 total 63.98 last 484.66) - open/close 29287 (time 1713279805.38 total 73.98 last 443.94) - open/close 34201 (time 1713279815.38 total 83.98 last 491.33) - open/close 38082 (time 1713279825.38 total 93.98 last 388.05) - open/close 40000 (time 1713279829.79 total 98.40 last 434.58) - open/close 44009 (time 1713279839.80 total 108.40 last 400.86) - open/close 48392 (time 1713279849.80 total 118.40 last 438.29) - open/close 50000 (time 1713279853.24 total 121.84 last 466.80) - open/close 53888 (time 1713279863.24 total 131.84 last 388.75) - open/close 58070 (time 1713279873.24 total 141.85 last 418.16) - open/close 60000 (time 1713279877.41 total 146.01 last 463.53) - open/close 64516 (time 1713279887.41 total 156.01 last 451.58) total: 64768 open/close in 156.66 seconds: 413.44 ops/second - /unlink 7099 (time 1713279899.23 total 10.00 last 709.83) - /unlink 10000 (time 1713279902.57 total 13.34 last 869.42) - /unlink 17763 (time 1713279912.57 total 23.34 last 776.20) - /unlink 20000 (time 1713279915.36 total 26.13 last 801.30) - /unlink 26284 (time 1713279925.37 total 36.13 last 628.33) - /unlink 30000 (time 1713279932.38 total 43.15 last 529.53) - /unlink 36527 (time 1713279942.38 total 53.15 last 652.66) - /unlink 40000 (time 1713279948.69 total 59.45 last 551.14) - /unlink 46159 (time 1713279958.69 total 69.45 last 615.76) - /unlink 50000 (time 1713279965.53 total 76.29 last 561.56) - /unlink 55342 (time 1713279975.53 total 86.29 last 534.15) - /unlink 60000 (time 1713279984.71 total 95.48 last 507.12) total: 64768 /unlink in 103.28 seconds: 627.11 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3103 (time 1713280023.02 total 10.00 last 310.24) - open/close 5798 (time 1713280033.02 total 20.00 last 269.46) - open/close 8418 (time 1713280043.02 total 30.01 last 261.92) - open/close 10000 (time 1713280047.66 total 34.64 last 341.31) - open/close 13404 (time 1713280057.66 total 44.64 last 340.39) - open/close 16866 (time 1713280067.66 total 54.64 last 346.20) - open/close 19918 (time 1713280077.66 total 64.64 last 305.15) - open/close 23689 (time 1713280087.66 total 74.64 last 377.07) - open/close 28712 (time 1713280097.66 total 84.65 last 502.28) - open/close 30000 (time 1713280100.75 total 87.74 last 416.82) - open/close 33851 (time 1713280110.75 total 97.74 last 385.09) - open/close 37165 (time 1713280120.75 total 107.74 last 331.39) - open/close 40000 (time 1713280130.34 total 117.32 last 295.72) - open/close 43057 (time 1713280140.34 total 127.32 last 305.64) - open/close 46536 (time 1713280150.34 total 137.33 last 347.81) - open/close 50000 (time 1713280159.64 total 146.62 last 372.68) - open/close 54811 (time 1713280169.64 total 156.62 last 481.03) - open/close 59210 (time 1713280179.64 total 166.62 last 439.83) - open/close 64203 (time 1713280189.64 total 176.62 last 499.29) total: 64768 open/close in 177.75 seconds: 364.37 ops/second - /unlink 5939 (time 1713280201.61 total 10.00 last 593.90) - /unlink 10000 (time 1713280210.36 total 18.75 last 463.95) - /unlink 14873 (time 1713280220.36 total 28.75 last 487.30) - /unlink 20000 (time 1713280230.30 total 38.69 last 516.12) - /unlink 25130 (time 1713280240.30 total 48.69 last 512.97) - /unlink 29788 (time 1713280250.30 total 58.69 last 465.78) - /unlink 34633 (time 1713280260.30 total 68.69 last 484.47) - /unlink 39760 (time 1713280270.30 total 78.69 last 512.66) - /unlink 46752 (time 1713280280.30 total 88.69 last 699.14) - /unlink 50000 (time 1713280286.14 total 94.53 last 556.16) - /unlink 54834 (time 1713280296.14 total 104.53 last 483.39) - /unlink 59904 (time 1713280306.14 total 114.53 last 506.99) total: 64768 /unlink in 123.91 seconds: 522.71 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server unloading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2