== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:31:28 (1713364288) Stopping clients: oleg139-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg139-client.virtnet /mnt/lustre2 (opts:-f) oleg139-server: oleg139-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg139-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3956 (time 1713364334.79 total 10.00 last 395.55) - open/close 8794 (time 1713364344.79 total 20.00 last 483.74) - open/close 10000 (time 1713364347.04 total 22.26 last 534.99) - open/close 14674 (time 1713364357.05 total 32.26 last 467.26) - open/close 20000 (time 1713364366.89 total 42.10 last 541.34) - open/close 24958 (time 1713364376.89 total 52.10 last 495.79) - open/close 29211 (time 1713364386.89 total 62.10 last 425.26) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 68.25 seconds: 477.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8348 (time 1713364404.14 total 10.00 last 834.80) - /unlink 10000 (time 1713364406.35 total 12.21 last 746.03) - /unlink 19200 (time 1713364416.35 total 22.21 last 919.97) - /unlink 25588 (time 1713364426.35 total 32.22 last 638.74) - /unlink 30000 (time 1713364432.64 total 38.50 last 701.92) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 43.06 seconds: 756.24 ops/second Waiting for MDT destroys to complete - open/close 5130 (time 1713364467.11 total 10.00 last 512.95) - open/close 10000 (time 1713364476.17 total 19.06 last 537.51) - open/close 15431 (time 1713364486.17 total 29.06 last 543.06) - open/close 20000 (time 1713364495.16 total 38.05 last 508.13) - open/close 26295 (time 1713364505.16 total 48.06 last 629.43) - open/close 30000 (time 1713364511.81 total 54.70 last 557.50) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.25 seconds: 549.51 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9568 (time 1713364527.21 total 10.00 last 956.78) - /unlink 19155 (time 1713364537.21 total 20.00 last 958.54) - /unlink 27781 (time 1713364547.21 total 30.00 last 862.58) - /unlink 30000 (time 1713364549.64 total 32.43 last 915.58) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 35.53 seconds: 916.49 ops/second Waiting for MDT destroys to complete - open/close 5567 (time 1713364590.16 total 10.00 last 556.64) - open/close 10000 (time 1713364597.16 total 17.00 last 633.20) - open/close 15710 (time 1713364607.16 total 27.00 last 570.97) - open/close 20000 (time 1713364614.58 total 34.42 last 578.66) - open/close 25945 (time 1713364624.58 total 44.42 last 594.46) - open/close 30000 (time 1713364631.16 total 50.99 last 616.47) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 55.79 seconds: 583.58 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9480 (time 1713364646.84 total 10.00 last 947.94) - /unlink 18788 (time 1713364656.84 total 20.00 last 930.76) - /unlink 27820 (time 1713364666.84 total 30.00 last 903.16) - /unlink 30000 (time 1713364669.06 total 32.22 last 983.60) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.81 seconds: 935.28 ops/second Waiting for MDT destroys to complete - open/close 5313 (time 1713364707.40 total 10.00 last 531.27) - open/close 10000 (time 1713364716.36 total 18.96 last 523.37) - open/close 15006 (time 1713364726.36 total 28.96 last 500.57) - open/close 20000 (time 1713364736.00 total 38.60 last 517.96) - open/close 25429 (time 1713364746.00 total 48.60 last 542.88) - open/close 30000 (time 1713364754.99 total 57.59 last 508.57) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.89 seconds: 526.11 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1713364769.30 total 9.18 last 1089.79) - /unlink 20000 (time 1713364778.01 total 17.88 last 1148.92) - /unlink 30000 (time 1713364787.16 total 27.04 last 1092.16) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 29.32 seconds: 1110.35 ops/second Waiting for MDT destroys to complete - open/close 5419 (time 1713364832.60 total 10.00 last 541.83) - open/close 10000 (time 1713364841.44 total 18.84 last 518.56) - open/close 15360 (time 1713364851.44 total 28.84 last 535.93) - open/close 20000 (time 1713364859.74 total 37.14 last 558.58) - open/close 25539 (time 1713364869.74 total 47.14 last 553.90) - open/close 30000 (time 1713364877.78 total 55.18 last 555.37) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.94 seconds: 543.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9594 (time 1713364893.40 total 10.00 last 959.34) - /unlink 19020 (time 1713364903.40 total 20.00 last 942.52) - /unlink 29161 (time 1713364913.41 total 30.00 last 1014.07) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.31 seconds: 977.62 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg139-server unloading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2