-----============= acceptance-small: replay-single ============----- Thu Apr 18 18:50:54 EDT 2024 excepting tests: 110f 131b 59 36 === replay-single: start setup 18:50:58 (1713480658) === oleg255-client.virtnet: executing check_config_client /mnt/lustre oleg255-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg255-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a509000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a509000.idle_timeout=debug disable quota as required oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === replay-single: finish setup 18:51:06 (1713480666) === debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 0a: empty replay =================== 18:51:07 (1713480667) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:51:09 (1713480669) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:51:22 (1713480682) targets are mounted 18:51:22 (1713480682) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 18:51:28 (1713480688) Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 18:51:29 (1713480689) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 18:51:42 (1713480702) targets are mounted 18:51:42 (1713480702) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.08 seconds: 252.14 ops/second - unlinked 0 (time 1713480706 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 0c: check replay-barrier =========== 18:51:48 (1713480708) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:51:51 (1713480711) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:52:03 (1713480723) targets are mounted 18:52:03 (1713480723) facet_failover done Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (78s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 0d: expired recovery with no clients ========================================================== 18:53:08 (1713480788) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:53:11 (1713480791) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:53:24 (1713480804) targets are mounted 18:53:24 (1713480804) facet_failover done Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre PASS 0d (79s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 1: simple create =================== 18:54:29 (1713480869) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:54:33 (1713480873) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:54:46 (1713480886) targets are mounted 18:54:46 (1713480886) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2a: touch ========================== 18:54:54 (1713480894) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:54:57 (1713480897) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:55:11 (1713480911) targets are mounted 18:55:11 (1713480911) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2b: touch ========================== 18:55:18 (1713480918) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:55:21 (1713480921) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:55:34 (1713480934) targets are mounted 18:55:34 (1713480934) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2c: setstripe replay =============== 18:55:41 (1713480941) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:55:44 (1713480944) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:55:57 (1713480957) targets are mounted 18:55:57 (1713480957) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2d: setdirstripe replay ============ 18:56:04 (1713480964) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:56:06 (1713480966) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:56:19 (1713480979) targets are mounted 18:56:19 (1713480979) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2e: O_CREAT|O_EXCL create replay === 18:56:24 (1713480984) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:56:28 (1713480988) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:56:42 (1713481002) targets are mounted 18:56:42 (1713481002) facet_failover done Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 18:56:48 (1713481008) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:56:51 (1713481011) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:57:03 (1713481023) targets are mounted 18:57:03 (1713481023) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 3b: replay failed open -ENOMEM ===== 18:57:09 (1713481029) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:57:11 (1713481031) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:57:24 (1713481044) targets are mounted 18:57:24 (1713481044) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 3c: replay failed open -ENOMEM ===== 18:57:30 (1713481050) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:57:32 (1713481052) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:57:45 (1713481065) targets are mounted 18:57:45 (1713481065) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 18:57:50 (1713481070) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:57:53 (1713481073) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:58:05 (1713481085) targets are mounted 18:58:05 (1713481085) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 4b: |x| rm 10 files ================ 18:58:11 (1713481091) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3761152 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7522304 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:58:13 (1713481093) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:58:26 (1713481106) targets are mounted 18:58:26 (1713481106) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 5: |x| 220 open(O_CREAT) =========== 18:58:31 (1713481111) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:58:35 (1713481115) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:58:47 (1713481127) targets are mounted 18:58:47 (1713481127) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 6a: mkdir + contained create ======= 18:58:59 (1713481139) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 8192 7530496 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:59:01 (1713481141) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:59:13 (1713481153) targets are mounted 18:59:13 (1713481153) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 6b: |X| rmdir ====================== 18:59:21 (1713481161) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:59:24 (1713481164) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:59:37 (1713481177) targets are mounted 18:59:37 (1713481177) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 7: mkdir |X| contained create ====== 18:59:43 (1713481183) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 18:59:45 (1713481185) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 18:59:57 (1713481197) targets are mounted 18:59:57 (1713481197) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 8: creat open |X| close ============ 19:00:03 (1713481203) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:00:05 (1713481205) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:00:17 (1713481217) targets are mounted 19:00:17 (1713481217) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 9: |X| create (same inum/gen) ====== 19:00:23 (1713481223) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:00:26 (1713481226) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:00:38 (1713481238) targets are mounted 19:00:38 (1713481238) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 10: create |X| rename unlink ======= 19:00:44 (1713481244) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:00:46 (1713481246) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:00:58 (1713481258) targets are mounted 19:00:58 (1713481258) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 19:01:04 (1713481264) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre new old Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:01:06 (1713481266) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:01:19 (1713481279) targets are mounted 19:01:19 (1713481279) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 12: open, unlink |X| close ========= 19:01:25 (1713481285) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:01:27 (1713481287) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:01:39 (1713481299) targets are mounted 19:01:39 (1713481299) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 13: open chmod 0 |x| write close === 19:01:45 (1713481305) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:01:47 (1713481307) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:01:59 (1713481319) targets are mounted 19:01:59 (1713481319) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 19:02:05 (1713481325) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:02:07 (1713481327) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:02:19 (1713481339) targets are mounted 19:02:19 (1713481339) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 19:02:27 (1713481347) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:02:30 (1713481350) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:02:44 (1713481364) targets are mounted 19:02:44 (1713481364) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 19:02:52 (1713481372) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:02:55 (1713481375) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:03:09 (1713481389) targets are mounted 19:03:09 (1713481389) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 19:03:16 (1713481396) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:03:20 (1713481400) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:03:33 (1713481413) targets are mounted 19:03:33 (1713481413) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 19:03:41 (1713481421) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 pid: 21665 will close Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:03:44 (1713481424) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:03:58 (1713481438) targets are mounted 19:03:58 (1713481438) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 19: mcreate, open, write, rename === 19:04:05 (1713481445) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre old Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:04:09 (1713481449) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:04:22 (1713481462) targets are mounted 19:04:22 (1713481462) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:04:30 (1713481470) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:04:33 (1713481473) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:04:47 (1713481487) targets are mounted 19:04:47 (1713481487) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 19:04:54 (1713481494) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1090 0x442 0x240000400 10000+0 records in 10000+0 records out 40960000 bytes (41 MB) copied, 1.26972 s, 32.3 MB/s Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:04:59 (1713481499) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:05:13 (1713481513) targets are mounted 19:05:13 (1713481513) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg255-server: oleg255-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg255-server: *.lustre-MDT0000.recovery_status status: COMPLETE sleep 5 for ZFS zfs Waiting for MDT destroys to complete before 6144, after 6144 PASS 20b (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 20c: check that client eviction does not affect file content ========================================================== 19:05:28 (1713481528) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 -rw-r--r-- 1 root root 1 Apr 18 19:05 /mnt/lustre/f20c.replay-single pdsh@oleg255-client: oleg255-client: ssh exited with exit code 5 PASS 20c (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 19:05:34 (1713481534) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7530496 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:05:36 (1713481536) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:05:50 (1713481550) targets are mounted 19:05:50 (1713481550) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:05:57 (1713481557) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7530496 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:06:00 (1713481560) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:06:14 (1713481574) targets are mounted 19:06:14 (1713481574) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 19:06:22 (1713481582) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:06:25 (1713481585) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:06:38 (1713481598) targets are mounted 19:06:38 (1713481598) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 19:06:46 (1713481606) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:06:50 (1713481610) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:07:03 (1713481623) targets are mounted 19:07:03 (1713481623) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:07:10 (1713481630) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:07:14 (1713481634) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:07:27 (1713481647) targets are mounted 19:07:27 (1713481647) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 19:07:35 (1713481655) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:07:38 (1713481658) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:07:52 (1713481672) targets are mounted 19:07:52 (1713481672) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 19:07:59 (1713481679) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:08:03 (1713481683) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:08:16 (1713481696) targets are mounted 19:08:16 (1713481696) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 19:08:23 (1713481703) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:08:27 (1713481707) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:08:40 (1713481720) targets are mounted 19:08:40 (1713481720) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 19:08:46 (1713481726) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:08:49 (1713481729) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:09:02 (1713481742) targets are mounted 19:09:02 (1713481742) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 19:09:10 (1713481750) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:09:13 (1713481753) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:09:26 (1713481766) targets are mounted 19:09:26 (1713481766) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 19:09:32 (1713481772) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:09:36 (1713481776) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:09:49 (1713481789) targets are mounted 19:09:49 (1713481789) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 19:09:57 (1713481797) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 5 PASS 32 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 19:10:04 (1713481804) total: 10 open/close in 0.04 seconds: 251.62 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Failover mds1 to oleg255-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.06 seconds: 157.07 ops/second PASS 33a (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 33b: test fid seq allocation ======= 19:10:18 (1713481818) fail_loc=0x1311 total: 10 open/close in 0.05 seconds: 192.89 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Failover mds1 to oleg255-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 total: 10 open/close in 0.07 seconds: 134.29 ops/second PASS 33b (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 19:10:31 (1713481831) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Failover mds1 to oleg255-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (10s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 35: test recovery from llog for unlink op ========================================================== 19:10:43 (1713481843) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Failover mds1 to oleg255-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (12s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-single test_36 skipping ALWAYS excluded test 36 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 19:10:57 (1713481857) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3456 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Failover mds1 to oleg255-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 37 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 19:11:11 (1713481871) total: 800 open/close in 4.32 seconds: 185.28 ops/second - unlinked 0 (time 1713481878 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:11:23 (1713481883) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:11:36 (1713481896) targets are mounted 19:11:36 (1713481896) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1713481900 ; total 0 ; last 0) total: 400 unlinks in 3 seconds: 133.333328 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (36s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 19:11:49 (1713481909) total: 800 open/close in 3.01 seconds: 265.76 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1713481915 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:11:58 (1713481918) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:12:11 (1713481931) targets are mounted 19:12:11 (1713481931) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1713481936 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 19:12:24 (1713481944) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0014288 s, 2.9 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00374269 s, 1.1 MB/s PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 42: recovery after ost failure ===== 19:12:28 (1713481948) total: 800 open/close in 3.52 seconds: 227.05 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3840 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1713481955 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second debug=-1 Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:12:38 (1713481958) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 19:12:52 (1713481972) targets are mounted 19:12:52 (1713481972) facet_failover done wait for MDS to timeout and recover debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck - unlinked 0 (time 1713482013 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (69s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 19:13:39 (1713482019) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:13:42 (1713482022) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:13:56 (1713482036) targets are mounted 19:13:56 (1713482036) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (32s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 44a: race in target handle connect ========================================================== 19:14:13 (1713482053) at_max=40 1 of 10 (1713482055) service : cur 5 worst 5 (at 1713482036, 19s ago) 4 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 2 of 10 (1713482060) service : cur 5 worst 5 (at 1713482036, 24s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 3 of 10 (1713482066) service : cur 5 worst 5 (at 1713482036, 30s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 4 of 10 (1713482072) service : cur 5 worst 5 (at 1713482036, 36s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 5 of 10 (1713482077) service : cur 5 worst 5 (at 1713482036, 41s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 6 of 10 (1713482083) service : cur 5 worst 5 (at 1713482036, 47s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 7 of 10 (1713482088) service : cur 5 worst 5 (at 1713482036, 53s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 8 of 10 (1713482094) service : cur 5 worst 5 (at 1713482036, 58s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 9 of 10 (1713482100) service : cur 5 worst 5 (at 1713482036, 64s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 10 of 10 (1713482105) service : cur 5 worst 5 (at 1713482036, 70s ago) 5 0 0 0 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (60s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 44b: race in target handle connect ========================================================== 19:15:16 (1713482116) 1 of 10 (1713482116) service : cur 5 worst 5 (at 1713482036, 80s ago) 5 0 0 0 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.155@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre 2 of 10 (1713482157) service : cur 40 worst 40 (at 1713482157, 0s ago) 40 0 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress 3 of 10 (1713482178) service : cur 40 worst 40 (at 1713482157, 21s ago) 40 0 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress 4 of 10 (1713482198) service : cur 40 worst 40 (at 1713482157, 41s ago) 40 40 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress 5 of 10 (1713482219) service : cur 40 worst 40 (at 1713482157, 62s ago) 40 40 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress 6 of 10 (1713482239) service : cur 40 worst 40 (at 1713482157, 83s ago) 40 40 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress 7 of 10 (1713482260) service : cur 40 worst 40 (at 1713482157, 103s ago) 40 40 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress 8 of 10 (1713482281) service : cur 40 worst 40 (at 1713482157, 124s ago) 40 40 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress 9 of 10 (1713482301) service : cur 40 worst 40 (at 1713482157, 145s ago) 40 40 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress 10 of 10 (1713482322) service : cur 40 worst 40 (at 1713482157, 165s ago) 40 40 0 0 fail_loc=0x80000704 error: recover: Connection timed out df: '/mnt/lustre': Operation already in progress PASS 44b (228s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 44c: race in target handle connect ========================================================== 19:19:06 (1713482346) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 100 create in 0.42 seconds: 235.68 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Failover mds1 to oleg255-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg255-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg255-client: oleg255-client: ssh exited with exit code 5 first stat failed: 5 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:19:19 (1713482359) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:19:33 (1713482373) targets are mounted 19:19:33 (1713482373) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (32s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 45: Handle failed close ============ 19:19:41 (1713482381) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 /mnt/lustre/f45.replay-single has type file OK PASS 45 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 19:19:46 (1713482386) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:20:05 (1713482405) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:20:18 (1713482418) targets are mounted 19:20:18 (1713482418) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (38s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 19:20:26 (1713482426) total: 20 open/close in 0.15 seconds: 136.35 ops/second Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:20:28 (1713482428) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 19:20:42 (1713482442) targets are mounted 19:20:42 (1713482442) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.12 seconds: 163.90 ops/second - unlinked 0 (time 1713482508 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (83s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 19:21:51 (1713482511) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 20 open/close in 0.14 seconds: 140.38 ops/second Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:21:55 (1713482515) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:22:07 (1713482527) targets are mounted 19:22:07 (1713482527) facet_failover done fail_loc=0x80000216 total: 20 open/close in 0.14 seconds: 141.55 ops/second - unlinked 0 (time 1713482589 ; total 0 ; last 0) total: 40 unlinks in 0 seconds: inf unlinks/second PASS 48 (80s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 19:23:13 (1713482593) PASS 50 (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 52: time out lock replay (3764) ==== 19:23:23 (1713482603) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.6896 fail_loc=0x80000157 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:23:26 (1713482606) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:23:39 (1713482619) targets are mounted 19:23:39 (1713482619) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (36s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 19:24:01 (1713482641) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:24:06 (1713482646) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:24:20 (1713482660) targets are mounted 19:24:20 (1713482660) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 19:24:28 (1713482668) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6896 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:24:33 (1713482673) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:24:47 (1713482687) targets are mounted 19:24:47 (1713482687) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 19:24:54 (1713482694) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:24:59 (1713482699) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:25:12 (1713482712) targets are mounted 19:25:12 (1713482712) facet_failover done fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 19:25:19 (1713482719) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:25:22 (1713482722) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:25:36 (1713482736) targets are mounted 19:25:36 (1713482736) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 19:25:44 (1713482744) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:25:49 (1713482749) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:26:03 (1713482763) targets are mounted 19:26:03 (1713482763) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 19:26:10 (1713482770) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:26:15 (1713482775) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:26:29 (1713482789) targets are mounted 19:26:29 (1713482789) facet_failover done fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 19:26:35 (1713482795) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:26:41 (1713482801) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:26:54 (1713482814) targets are mounted 19:26:54 (1713482814) facet_failover done /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (31s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 19:27:08 (1713482828) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:27:14 (1713482834) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:27:27 (1713482847) targets are mounted 19:27:27 (1713482847) facet_failover done fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (31s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 19:27:41 (1713482861) fail_loc=0x8000012b fail_loc=0x0 rm: cannot remove '/mnt/lustre/f55.replay-single'touch: cannot touch '/mnt/lustre/f55.replay-single': No such file or directory : No such file or directory PASS 55 (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 19:28:02 (1713482882) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:28:05 (1713482885) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:28:19 (1713482899) targets are mounted 19:28:19 (1713482899) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (32s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 57: test recovery from llog for setattr op ========================================================== 19:28:36 (1713482916) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:28:40 (1713482920) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:28:54 (1713482934) targets are mounted 19:28:54 (1713482934) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg255-server: oleg255-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg255-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg255-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 19:29:05 (1713482945) fail_loc=0x8000012c total: 2500 open/close in 9.84 seconds: 254.16 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:29:20 (1713482960) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:29:34 (1713482974) targets are mounted 19:29:34 (1713482974) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1713482993 ; total 0 ; last 0) total: 2500 unlinks in 4 seconds: 625.000000 unlinks/second PASS 58a (54s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 58b: test replay of setxattr op ==== 19:30:01 (1713483001) Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:30:04 (1713483004) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:30:18 (1713483018) targets are mounted 19:30:18 (1713483018) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) oleg255-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 19:30:27 (1713483027) Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) PASS 58c (41s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-single test_59 skipping ALWAYS excluded test 59 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 19:31:11 (1713483071) total: 200 open/close in 0.82 seconds: 242.76 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1713483074 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:31:16 (1713483076) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:31:29 (1713483089) targets are mounted 19:31:29 (1713483089) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1713483093 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second PASS 60 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 19:31:37 (1713483097) total: 800 open/close in 3.20 seconds: 249.73 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre - unlinked 0 (time 1713483105 ; total 0 ; last 0) total: 800 unlinks in 1 seconds: 800.000000 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:31:49 (1713483109) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 19:32:03 (1713483123) targets are mounted 19:32:03 (1713483123) facet_failover done Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:32:14 (1713483134) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 19:32:28 (1713483148) targets are mounted 19:32:28 (1713483148) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (87s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 19:33:06 (1713483186) fail_loc=0x8000013a Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:33:08 (1713483188) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:33:22 (1713483202) targets are mounted 19:33:22 (1713483202) facet_failover done Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:33:33 (1713483213) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:33:47 (1713483227) targets are mounted 19:33:47 (1713483227) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00441998 s, 927 kB/s PASS 61b (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 19:33:55 (1713483235) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:34:07 (1713483247) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 19:34:21 (1713483261) targets are mounted 19:34:21 (1713483261) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (32s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 19:34:29 (1713483269) Stopping /mnt/lustre-mds1 (opts:) on oleg255-server fail_loc=0x80000605 Starting mgs: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg255-client: oleg255-server: ssh exited with exit code 95 Start of lustre-mdt1/mdt1 on mgs failed 95 fail_loc=0 Starting mgs: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 62: don't mis-drop resent replay === 19:34:42 (1713483282) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.14 seconds: 177.99 ops/second fail_loc=0x80000707 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:34:46 (1713483286) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:34:59 (1713483299) targets are mounted 19:34:59 (1713483299) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1713483320 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (39s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 65a: AT: verify early replies ====== 19:35:24 (1713483324) at_history=8 at_history=8 debug=other fail_val=11000 fail_loc=0x8000050a 00000100:00001000:0.0:1713483354.254365:0:7142:0:(client.c:537:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff8800a8dcc000 x1796714639562880/t0(0) o101->lustre-MDT0000-mdc-ffff8800b3c8d000@192.168.202.155@tcp:12/10 lens 664/66320 e 1 to 0 dl 1713483389 ref 2 fl Rpc:PQr/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 portal 12 : cur 36 worst 40 (at 1713482157, 1202s ago) 36 5 5 5 portal 23 : cur 5 worst 10 (at 1713480909, 2450s ago) 10 5 5 10 portal 30 : cur 5 worst 5 (at 1713480870, 2489s ago) 5 0 0 0 portal 17 : cur 5 worst 5 (at 1713480943, 2416s ago) 5 5 5 5 portal 13 : cur 10 worst 10 (at 1713481259, 2100s ago) 10 10 0 0 portal 12 : cur 5 worst 40 (at 1713482157, 1211s ago) 36 5 5 5 portal 23 : cur 5 worst 10 (at 1713480909, 2459s ago) 10 5 5 10 portal 30 : cur 5 worst 5 (at 1713480870, 2498s ago) 5 0 0 0 portal 17 : cur 5 worst 5 (at 1713480943, 2425s ago) 5 5 5 5 portal 13 : cur 10 worst 10 (at 1713481259, 2109s ago) 10 10 0 0 PASS 65a (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 19:36:12 (1713483372) at_history=8 at_history=8 debug=other trace fail_val=11 fail_loc=0x224 fail_loc=0 00000100:00001000:0.0:1713483402.494757:0:1907:0:(client.c:537:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff8800a937df80 x1796714639572224/t0(0) o4->lustre-OST0000-osc-ffff8800b3c8d000@192.168.202.155@tcp:6/4 lens 4584/448 e 1 to 0 dl 1713483437 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck portal 28 : cur 5 worst 5 (at 1713480864, 2544s ago) 5 5 5 5 portal 7 : cur 5 worst 5 (at 1713480866, 2542s ago) 5 5 5 5 portal 17 : cur 5 worst 5 (at 1713480939, 2469s ago) 5 5 0 5 portal 6 : cur 36 worst 36 (at 1713483407, 1s ago) 36 0 0 0 PASS 65b (37s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 19:36:52 (1713483412) at_history=8 at_history=8 portal 12 : cur 5 worst 40 (at 1713482157, 1277s ago) 5 36 5 5 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 5 worst 40 (at 1713482157, 1283s ago) 5 36 5 5 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 36 worst 40 (at 1713482157, 1294s ago) 36 36 5 5 fail_loc=0 portal 12 : cur 5 worst 40 (at 1713482157, 1303s ago) 36 36 5 5 Current MDT timeout 5, worst 40 PASS 66a (51s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 66b: AT: verify net latency adjusts ========================================================== 19:37:44 (1713483464) at_history=8 at_history=8 fail_val=10 fail_loc=0x50c fail_loc=0 network timeout orig 5, cur 10, worst 10 PASS 66b (85s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 19:39:11 (1713483551) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (75s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 19:40:28 (1713483628) at_history=8 at_history=8 Creating to objid 5441 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 18 open/close in 0.09 seconds: 191.81 ops/second Connected clients: oleg255-client.virtnet oleg255-client.virtnet service : cur 5 worst 5 (at 1713480634, 3019s ago) 1 0 0 1 phase 2 0 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg255-client.virtnet oleg255-client.virtnet service : cur 5 worst 5 (at 1713480634, 3020s ago) 1 1 0 0 0 osc reconnect attempts on 2nd slow PASS 67b (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 68: AT: verify slowing locks ======= 19:40:58 (1713483658) at_history=8 at_history=8 /home/green/git/lustre-release/lustre/tests/replay-single.sh: line 1968: $ldlm_enqueue_min: ambiguous redirect fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 /home/green/git/lustre-release/lustre/tests/replay-single.sh: line 1983: $ldlm_enqueue_min: ambiguous redirect PASS 68 (70s) debug_raw_pointers=0 debug_raw_pointers=0 Cleaning up AT ... debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70a: check multi client t-f ======== 19:42:11 (1713483731) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 19:42:14 (1713483734) Starting client oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg255-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14205 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3727360 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3682304 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7409664 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:42:22 (1713483742) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:42:35 (1713483755) targets are mounted 19:42:35 (1713483755) facet_failover done oleg255-client.virtnet: looking for dbench program oleg255-client.virtnet: /usr/bin/dbench oleg255-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg255-client.virtnet oleg255-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg255-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg255-client.virtnet' oleg255-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg255-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg255-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg255-client.virtnet at Thu Apr 18 19:42:15 EDT 2024 oleg255-client.virtnet: waiting for dbench pid 14230 oleg255-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg255-client.virtnet: oleg255-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg255-client.virtnet: failed to create barrier semaphore oleg255-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg255-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg255-client.virtnet: releasing clients oleg255-client.virtnet: 1 313 11.48 MB/sec warmup 1 sec latency 23.489 ms oleg255-client.virtnet: 1 664 10.58 MB/sec warmup 2 sec latency 30.110 ms oleg255-client.virtnet: 1 876 7.22 MB/sec warmup 3 sec latency 87.426 ms oleg255-client.virtnet: 1 1177 5.75 MB/sec warmup 4 sec latency 39.754 ms oleg255-client.virtnet: 1 1558 5.22 MB/sec warmup 5 sec latency 44.816 ms oleg255-client.virtnet: 1 1787 4.38 MB/sec warmup 6 sec latency 306.703 ms oleg255-client.virtnet: 1 1787 3.76 MB/sec warmup 7 sec latency 1306.996 ms oleg255-client.virtnet: 1 1787 3.29 MB/sec warmup 8 sec latency 2307.248 ms oleg255-client.virtnet: 1 1787 2.92 MB/sec warmup 9 sec latency 3307.557 ms oleg255-client.virtnet: 1 1787 2.63 MB/sec warmup 10 sec latency 4307.915 ms oleg255-client.virtnet: 1 1787 2.39 MB/sec warmup 11 sec latency 5308.203 ms oleg255-client.virtnet: 1 1787 2.19 MB/sec warmup 12 sec latency 6308.434 ms oleg255-client.virtnet: 1 1787 2.02 MB/sec warmup 13 sec latency 7308.678 ms oleg255-client.virtnet: 1 1787 1.88 MB/sec warmup 14 sec latency 8308.979 ms oleg255-client.virtnet: 1 1787 1.75 MB/sec warmup 15 sec latency 9309.321 ms oleg255-client.virtnet: 1 1787 1.64 MB/sec warmup 16 sec latency 10309.632 ms oleg255-client.virtnet: 1 1787 1.55 MB/sec warmup 17 sec latency 11309.913 ms oleg255-client.virtnet: 1 1787 1.46 MB/sec warmup 18 sec latency 12310.133 ms oleg255-client.virtnet: 1 1787 1.38 MB/sec warmup 19 sec latency 13310.374 ms oleg255-client.virtnet: 1 1787 1.31 MB/sec warmup 20 sec latency 14310.636 ms oleg255-client.virtnet: 1 1787 1.25 MB/sec warmup 21 sec latency 15310.965 ms oleg255-client.virtnet: 1 1787 1.20 MB/sec warmup 22 sec latency 16311.206 ms oleg255-client.virtnet: 1 1787 1.14 MB/sec warmup 23 sec latency 17311.494 ms oleg255-client.virtnet: 1 1787 1.10 MB/sec warmup 24 sec latency 18311.907 ms oleg255-client.virtnet: 1 1787 1.05 MB/sec warmup 25 sec latency 19312.222 ms oleg255-client.virtnet: 1 1787 1.01 MB/sec warmup 26 sec latency 20312.518 ms oleg255-client.virtnet: 1 1787 0.97 MB/sec warmup 27 sec latency 21312.762 ms oleg255-client.virtnet: 1 1787 0.94 MB/sec warmup 28 sec latency 22313.102 ms oleg255-client.virtnet: 1 1787 0.91 MB/sec warmup 29 sec latency 23313.423 ms oleg255-client.virtnet: 1 1787 0.88 MB/sec warmup 30 sec latency 24313.772 ms oleg255-client.virtnet: 1 1787 0.85 MB/sec warmup 31 sec latency 25314.094 ms oleg255-client.virtnet: 1 1787 0.82 MB/sec warmup 32 sec oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3731456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3720192 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7451648 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:42:56 (1713483776) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:43:10 (1713483790) targets are mounted 19:43:10 (1713483790) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3732480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3714048 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7446528 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:43:21 (1713483801) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 latency 26314.343 ms oleg255-client.virtnet: 1 2018 0.80 MB/sec warmup 33 sec latency 26623.741 ms oleg255-client.virtnet: 1 2400 0.82 MB/sec warmup 34 sec latency 42.500 ms oleg255-client.virtnet: 1 2898 0.94 MB/sec warmup 35 sec latency 39.363 ms oleg255-client.virtnet: 1 3365 0.98 MB/sec warmup 36 sec latency 49.906 ms oleg255-client.virtnet: 1 3797 1.07 MB/sec warmup 37 sec latency 15.846 ms oleg255-client.virtnet: 1 4028 1.05 MB/sec warmup 38 sec latency 16.636 ms oleg255-client.virtnet: 1 4301 1.04 MB/sec warmup 39 sec latency 50.601 ms oleg255-client.virtnet: 1 4504 1.02 MB/sec warmup 40 sec latency 153.785 ms oleg255-client.virtnet: 1 4504 1.00 MB/sec warmup 41 sec latency 1153.985 ms oleg255-client.virtnet: 1 4504 0.97 MB/sec warmup 42 sec latency 2154.197 ms oleg255-client.virtnet: 1 4504 0.95 MB/sec warmup 43 sec latency 3154.430 ms oleg255-client.virtnet: 1 4504 0.93 MB/sec warmup 44 sec latency 4154.745 ms oleg255-client.virtnet: 1 4504 0.91 MB/sec warmup 45 sec latency 5155.030 ms oleg255-client.virtnet: 1 4504 0.89 MB/sec warmup 46 sec latency 6155.304 ms oleg255-client.virtnet: 1 4504 0.87 MB/sec warmup 47 sec latency 7155.548 ms oleg255-client.virtnet: 1 4504 0.85 MB/sec warmup 48 sec latency 8155.803 ms oleg255-client.virtnet: 1 4504 0.83 MB/sec warmup 49 sec latency 9156.051 ms oleg255-client.virtnet: 1 4504 0.82 MB/sec warmup 50 sec latency 10156.289 ms oleg255-client.virtnet: 1 4504 0.80 MB/sec warmup 51 sec latency 11156.587 ms oleg255-client.virtnet: 1 4504 0.79 MB/sec warmup 52 sec latency 12156.805 ms oleg255-client.virtnet: 1 4504 0.77 MB/sec warmup 53 sec latency 13157.005 ms oleg255-client.virtnet: 1 4504 0.76 MB/sec warmup 54 sec latency 14157.206 ms oleg255-client.virtnet: 1 4504 0.74 MB/sec warmup 55 sec latency 15157.432 ms oleg255-client.virtnet: 1 4504 0.73 MB/sec warmup 56 sec latency 16157.618 ms oleg255-client.virtnet: 1 4504 0.72 MB/sec warmup 57 sec latency 17157.824 ms oleg255-client.virtnet: 1 4706 0.73 MB/sec warmup 58 sec latency 17611.827 ms oleg255-client.virtnet: 1 5098 0.77 MB/sec warmup 59 sec latency 55.851 ms oleg255-client.virtnet: 1 5730 0.40 MB/sec execute 1 sec latency 37.922 ms oleg255-client.virtnet: 1 6044 1.19 MB/sec execute 2 sec latency 53.813 ms oleg255-client.virtnet: 1 6567 2.40 MB/sec execute 3 sec latency 42.164 ms oleg255-client.virtnet: 1 7171 3.07 MB/sec execute 4 sec latency 39.052 ms oleg255-client.virtnet: 1 7438 2.74 MB/sec execute 5 sec latency 36.607 ms oleg255-client.virtnet: 1 7438 2.29 MB/sec execute 6 sec latency 1036.841 ms oleg255-client.virtnet: 1 7438 1.96 MB/sec execute 7 sec latency 2037.025 ms oleg255-client.virtnet: 1 7438 1.71 MB/sec execute 8 sec latency 3037.222 ms oleg255-client.virtnet: 1 7438 1.52 MB/sec execute 9 sec latency 4037.453 ms oleg255-client.virtnet: 1 7438 1.37 MB/sec execute 10 sec latency 5037.646 ms oleg255-client.virtnet: 1 7438 1.25 MB/sec execute 11 sec latency 6037.880 ms oleg255-client.virtnet: 1 7438 1.14 MB/sec execute 12 sec latency 7038.070 ms oleg255-client.virtnet: 1 7438 1.06 MB/sec execute 13 sec latency 8038.294 ms oleg255-client.virtnet: 1 7438 0.98 MB/sec execute 14 sec latency 9038.488 ms oleg255-client.virtnet: 1 7438 0.91 MB/sec execute 15 sec latency 10038.656 ms oleg255-client.virtnet: 1 7438 0.86 MB/sec execute 16 sec latency 11038.900 ms oleg255-client.virtnet: 1 7438 0.81 MB/sec execute 17 sec latency 12039.108 ms oleg255-client.virtneoleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:43:35 (1713483815) targets are mounted 19:43:35 (1713483815) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3722240 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7468032 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:43:52 (1713483832) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:44:06 (1713483846) targets are mounted 19:44:06 (1713483846) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3742720 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3694592 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7437312 1% /mnt/lustre t: 1 7438 0.76 MB/sec execute 18 sec latency 13039.328 ms oleg255-client.virtnet: 1 7438 0.72 MB/sec execute 19 sec latency 14039.554 ms oleg255-client.virtnet: 1 7438 0.69 MB/sec execute 20 sec latency 15039.776 ms oleg255-client.virtnet: 1 7438 0.65 MB/sec execute 21 sec latency 16040.007 ms oleg255-client.virtnet: 1 7438 0.62 MB/sec execute 22 sec latency 17040.208 ms oleg255-client.virtnet: 1 7558 0.60 MB/sec execute 23 sec latency 17495.927 ms oleg255-client.virtnet: 1 7821 0.58 MB/sec execute 24 sec latency 61.839 ms oleg255-client.virtnet: 1 8025 0.58 MB/sec execute 25 sec latency 41.480 ms oleg255-client.virtnet: 1 8339 0.61 MB/sec execute 26 sec latency 43.026 ms oleg255-client.virtnet: 1 8721 0.70 MB/sec execute 27 sec latency 36.982 ms oleg255-client.virtnet: 1 9041 0.69 MB/sec execute 28 sec latency 43.428 ms oleg255-client.virtnet: 1 9451 0.71 MB/sec execute 29 sec latency 40.668 ms oleg255-client.virtnet: 1 9818 0.82 MB/sec execute 30 sec latency 56.017 ms oleg255-client.virtnet: 1 9818 0.80 MB/sec execute 31 sec latency 1056.209 ms oleg255-client.virtnet: 1 9818 0.77 MB/sec execute 32 sec latency 2056.436 ms oleg255-client.virtnet: 1 9818 0.75 MB/sec execute 33 sec latency 3056.714 ms oleg255-client.virtnet: 1 9818 0.73 MB/sec execute 34 sec latency 4056.956 ms oleg255-client.virtnet: 1 9818 0.71 MB/sec execute 35 sec latency 5057.136 ms oleg255-client.virtnet: 1 9818 0.69 MB/sec execute 36 sec latency 6057.350 ms oleg255-client.virtnet: 1 9818 0.67 MB/sec execute 37 sec latency 7057.567 ms oleg255-client.virtnet: 1 9818 0.65 MB/sec execute 38 sec latency 8057.796 ms oleg255-client.virtnet: 1 9818 0.63 MB/sec execute 39 sec latency 9058.004 ms oleg255-client.virtnet: 1 9818 0.62 MB/sec execute 40 sec latency 10058.192 ms oleg255-client.virtnet: 1 9818 0.60 MB/sec execute 41 sec latency 11058.361 ms oleg255-client.virtnet: 1 9818 0.59 MB/sec execute 42 sec latency 12058.547 ms oleg255-client.virtnet: 1 9818 0.57 MB/sec execute 43 sec latency 13058.763 ms oleg255-client.virtnet: 1 9818 0.56 MB/sec execute 44 sec latency 14058.929 ms oleg255-client.virtnet: 1 9818 0.55 MB/sec execute 45 sec latency 15059.081 ms oleg255-client.virtnet: 1 9818 0.54 MB/sec execute 46 sec latency 16059.251 ms oleg255-client.virtnet: 1 9818 0.53 MB/sec execute 47 sec latency 17059.402 ms oleg255-client.virtnet: 1 9818 0.51 MB/sec execute 48 sec latency 18059.561 ms oleg255-client.virtnet: 1 9818 0.50 MB/sec execute 49 sec latency 19059.754 ms oleg255-client.virtnet: 1 9818 0.49 MB/sec execute 50 sec latency 20059.958 ms oleg255-client.virtnet: 1 9818 0.48 MB/sec execute 51 sec latency 21060.175 ms oleg255-client.virtnet: 1 9818 0.48 MB/sec execute 52 sec latency 22060.403 ms oleg255-client.virtnet: 1 9818 0.47 MB/sec execute 53 sec latency 23060.645 ms oleg255-client.virtnet: 1 9818 0.46 MB/sec execute 54 sec latency 24060.872 ms oleg255-client.virtnet: 1 9818 0.45 MB/sec execute 55 sec latency 25061.096 ms oleg255-client.virtnet: 1 9818 0.44 MB/sec execute 56 sec latency 26061.338 ms oleg255-client.virtnet: 1 9818 0.43 MB/sec execute 57 sec latency 27061.643 ms oleg255-client.virtnet: 1 10099 0.45 MB/sec execute 58 sec latency 27427.103 ms oleg255-client.virtnet: 1 10666 0.53 MB/sec execute 59 sec latency 47.893 ms oleg255-client.virtnet: 1 10942 0.54 MB/sec execute 60 sec latency 18.749 ms oleg255-client.virtnet: 1 11169 0.54 MB/sec execute 61 sec latency 16.335 ms oleg255-client.virtntest_70b fail mds1 5 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:44:21 (1713483861) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:44:35 (1713483875) targets are mounted 19:44:35 (1713483875) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3716096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3709952 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7426048 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:44:46 (1713483886) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:45:00 (1713483900) targets are mounted 19:45:00 (1713483900) facet_failover done et: 1 11429 0.54 MB/sec execute 62 sec latency 71.465 ms oleg255-client.virtnet: 1 11658 0.54 MB/sec execute 63 sec latency 42.750 ms oleg255-client.virtnet: 1 12077 0.59 MB/sec execute 64 sec latency 16.459 ms oleg255-client.virtnet: 1 12342 0.59 MB/sec execute 65 sec latency 43.155 ms oleg255-client.virtnet: 1 12342 0.58 MB/sec execute 66 sec latency 1034.516 ms oleg255-client.virtnet: 1 12342 0.57 MB/sec execute 67 sec latency 2034.710 ms oleg255-client.virtnet: 1 12342 0.56 MB/sec execute 68 sec latency 3034.915 ms oleg255-client.virtnet: 1 12342 0.55 MB/sec execute 69 sec latency 4035.077 ms oleg255-client.virtnet: 1 12342 0.54 MB/sec execute 70 sec latency 5035.269 ms oleg255-client.virtnet: 1 12342 0.54 MB/sec execute 71 sec latency 6035.504 ms oleg255-client.virtnet: 1 12342 0.53 MB/sec execute 72 sec latency 7035.700 ms oleg255-client.virtnet: 1 12342 0.52 MB/sec execute 73 sec latency 8035.905 ms oleg255-client.virtnet: 1 12342 0.52 MB/sec execute 74 sec latency 9036.094 ms oleg255-client.virtnet: 1 12342 0.51 MB/sec execute 75 sec latency 10036.270 ms oleg255-client.virtnet: 1 12342 0.50 MB/sec execute 76 sec latency 11036.452 ms oleg255-client.virtnet: 1 12342 0.50 MB/sec execute 77 sec latency 12036.636 ms oleg255-client.virtnet: 1 12342 0.49 MB/sec execute 78 sec latency 13036.871 ms oleg255-client.virtnet: 1 12342 0.48 MB/sec execute 79 sec latency 14037.072 ms oleg255-client.virtnet: 1 12342 0.48 MB/sec execute 80 sec latency 15037.298 ms oleg255-client.virtnet: 1 12342 0.47 MB/sec execute 81 sec latency 16037.515 ms oleg255-client.virtnet: 1 12342 0.46 MB/sec execute 82 sec latency 17037.760 ms oleg255-client.virtnet: 1 12499 0.46 MB/sec execute 83 sec latency 17452.771 ms oleg255-client.virtnet: 1 12857 0.46 MB/sec execute 84 sec latency 52.517 ms oleg255-client.virtnet: 1 13150 0.48 MB/sec execute 85 sec latency 42.208 ms oleg255-client.virtnet: 1 13661 0.53 MB/sec execute 86 sec latency 46.765 ms oleg255-client.virtnet: 1 14229 0.58 MB/sec execute 87 sec latency 52.073 ms oleg255-client.virtnet: 1 14518 0.59 MB/sec execute 88 sec latency 15.866 ms oleg255-client.virtnet: 1 14762 0.58 MB/sec execute 89 sec latency 15.809 ms oleg255-client.virtnet: 1 15008 0.58 MB/sec execute 90 sec latency 54.015 ms oleg255-client.virtnet: 1 15009 0.58 MB/sec execute 91 sec latency 993.548 ms oleg255-client.virtnet: 1 15009 0.57 MB/sec execute 92 sec latency 1993.822 ms oleg255-client.virtnet: 1 15009 0.56 MB/sec execute 93 sec latency 2994.112 ms oleg255-client.virtnet: 1 15009 0.56 MB/sec execute 94 sec latency 3994.418 ms oleg255-client.virtnet: 1 15009 0.55 MB/sec execute 95 sec latency 4994.689 ms oleg255-client.virtnet: 1 15009 0.55 MB/sec execute 96 sec latency 5994.994 ms oleg255-client.virtnet: 1 15009 0.54 MB/sec execute 97 sec latency 6995.248 ms oleg255-client.virtnet: 1 15009 0.54 MB/sec execute 98 sec latency 7995.485 ms oleg255-client.virtnet: 1 15009 0.53 MB/sec execute 99 sec latency 8995.832 ms oleg255-client.virtnet: 1 15009 0.52 MB/sec execute 100 sec latency 9996.123 ms oleg255-client.virtnet: 1 15009 0.52 MB/sec execute 101 sec latency 10996.298 ms oleg255-client.virtnet: 1 15009 0.51 MB/sec execute 102 sec latency 11996.504 ms oleg255-client.virtnet: 1 15009 0.51 MB/sec execute 103 sec latency 13002.577 ms oleg255-client.virtnet: 1 15009 0.50 MB/sec execute 104 sec latency 14005.908 ms oleg255-client.virtnet: 1 15009 0.50 MB/sec execute 105 sec latency 15006.133 ms oleg255-client.virtnet: 1 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3742720 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3708928 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7451648 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:45:11 (1713483911) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:45:25 (1713483925) targets are mounted 19:45:25 (1713483925) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3720192 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7456768 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:45:36 (1713483936) shut down 15009 0.50 MB/sec execute 106 sec latency 16006.372 ms oleg255-client.virtnet: 1 15009 0.49 MB/sec execute 107 sec latency 17006.640 ms oleg255-client.virtnet: 1 15119 0.49 MB/sec execute 108 sec latency 17577.815 ms oleg255-client.virtnet: 1 15417 0.50 MB/sec execute 109 sec latency 54.702 ms oleg255-client.virtnet: 1 15775 0.52 MB/sec execute 110 sec latency 59.261 ms oleg255-client.virtnet: 1 16072 0.52 MB/sec execute 111 sec latency 44.686 ms oleg255-client.virtnet: 1 16447 0.52 MB/sec execute 112 sec latency 43.916 ms oleg255-client.virtnet: 1 16777 0.53 MB/sec execute 113 sec latency 58.545 ms oleg255-client.virtnet: 1 17383 0.57 MB/sec execute 114 sec latency 41.034 ms oleg255-client.virtnet: 1 17837 0.61 MB/sec execute 115 sec latency 37.450 ms oleg255-client.virtnet: 1 17872 0.60 MB/sec execute 116 sec latency 898.017 ms oleg255-client.virtnet: 1 17872 0.60 MB/sec execute 117 sec latency 1898.308 ms oleg255-client.virtnet: 1 17872 0.59 MB/sec execute 118 sec latency 2898.541 ms oleg255-client.virtnet: 1 17872 0.59 MB/sec execute 119 sec latency 3898.783 ms oleg255-client.virtnet: 1 17872 0.58 MB/sec execute 120 sec latency 4899.033 ms oleg255-client.virtnet: 1 17872 0.58 MB/sec execute 121 sec latency 5899.271 ms oleg255-client.virtnet: 1 17872 0.57 MB/sec execute 122 sec latency 6899.495 ms oleg255-client.virtnet: 1 17872 0.57 MB/sec execute 123 sec latency 7899.733 ms oleg255-client.virtnet: 1 17872 0.56 MB/sec execute 124 sec latency 8899.954 ms oleg255-client.virtnet: 1 17872 0.56 MB/sec execute 125 sec latency 9900.209 ms oleg255-client.virtnet: 1 17872 0.55 MB/sec execute 126 sec latency 10900.433 ms oleg255-client.virtnet: 1 17872 0.55 MB/sec execute 127 sec latency 11900.683 ms oleg255-client.virtnet: 1 17872 0.54 MB/sec execute 128 sec latency 12900.880 ms oleg255-client.virtnet: 1 17872 0.54 MB/sec execute 129 sec latency 13901.122 ms oleg255-client.virtnet: 1 17872 0.54 MB/sec execute 130 sec latency 14901.329 ms oleg255-client.virtnet: 1 17872 0.53 MB/sec execute 131 sec latency 15901.531 ms oleg255-client.virtnet: 1 17872 0.53 MB/sec execute 132 sec latency 16901.735 ms oleg255-client.virtnet: 1 17998 0.53 MB/sec execute 133 sec latency 17418.661 ms oleg255-client.virtnet: 1 18226 0.53 MB/sec execute 134 sec latency 15.980 ms oleg255-client.virtnet: 1 18472 0.53 MB/sec execute 135 sec latency 73.640 ms oleg255-client.virtnet: 1 18671 0.53 MB/sec execute 136 sec latency 43.952 ms oleg255-client.virtnet: 1 18992 0.53 MB/sec execute 137 sec latency 38.041 ms oleg255-client.virtnet: 1 19363 0.55 MB/sec execute 138 sec latency 31.269 ms oleg255-client.virtnet: 1 19639 0.55 MB/sec execute 139 sec latency 38.967 ms oleg255-client.virtnet: 1 20025 0.55 MB/sec execute 140 sec latency 29.212 ms oleg255-client.virtnet: 1 20041 0.55 MB/sec execute 141 sec latency 970.569 ms oleg255-client.virtnet: 1 20041 0.54 MB/sec execute 142 sec latency 1970.767 ms oleg255-client.virtnet: 1 20041 0.54 MB/sec execute 143 sec latency 2970.989 ms oleg255-client.virtnet: 1 20041 0.53 MB/sec execute 144 sec latency 3971.194 ms oleg255-client.virtnet: 1 20041 0.53 MB/sec execute 145 sec latency 4971.420 ms oleg255-client.virtnet: 1 20041 0.53 MB/sec execute 146 sec latency 5971.652 ms oleg255-client.virtnet: 1 20041 0.52 MB/sec execute 147 sec latency 6971.871 ms oleg255-client.virtnet: 1 20041 0.52 MB/sec execute 148 sec latency 7972.112 ms oleg255-client.virtnet: 1 20041 0.52 MB/sec execute 149 sec latency 8972.368 ms oleg255-client.virtnet: 1 20041 0.51 MB/sec Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:45:50 (1713483950) targets are mounted 19:45:50 (1713483950) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3734528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3705856 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7440384 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:46:01 (1713483961) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:46:15 (1713483975) targets are mounted 19:46:15 (1713483975) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3701760 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7446528 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:46:26 (1713483986) shut down execute 150 sec latency 9972.591 ms oleg255-client.virtnet: 1 20041 0.51 MB/sec execute 151 sec latency 10972.726 ms oleg255-client.virtnet: 1 20041 0.51 MB/sec execute 152 sec latency 11972.860 ms oleg255-client.virtnet: 1 20041 0.50 MB/sec execute 153 sec latency 12973.065 ms oleg255-client.virtnet: 1 20041 0.50 MB/sec execute 154 sec latency 13973.272 ms oleg255-client.virtnet: 1 20041 0.50 MB/sec execute 155 sec latency 14973.460 ms oleg255-client.virtnet: 1 20041 0.49 MB/sec execute 156 sec latency 15973.665 ms oleg255-client.virtnet: 1 20041 0.49 MB/sec execute 157 sec latency 16973.911 ms oleg255-client.virtnet: 1 20233 0.50 MB/sec execute 158 sec latency 17270.585 ms oleg255-client.virtnet: 1 20758 0.53 MB/sec execute 159 sec latency 39.996 ms oleg255-client.virtnet: 1 21305 0.55 MB/sec execute 160 sec latency 38.539 ms oleg255-client.virtnet: 1 21599 0.56 MB/sec execute 161 sec latency 16.996 ms oleg255-client.virtnet: 1 21831 0.56 MB/sec execute 162 sec latency 16.149 ms oleg255-client.virtnet: 1 22105 0.56 MB/sec execute 163 sec latency 52.802 ms oleg255-client.virtnet: 1 22317 0.56 MB/sec execute 164 sec latency 47.800 ms oleg255-client.virtnet: 1 22605 0.56 MB/sec execute 165 sec latency 158.893 ms oleg255-client.virtnet: 1 22605 0.56 MB/sec execute 166 sec latency 1159.085 ms oleg255-client.virtnet: 1 22605 0.55 MB/sec execute 167 sec latency 2159.285 ms oleg255-client.virtnet: 1 22605 0.55 MB/sec execute 168 sec latency 3159.459 ms oleg255-client.virtnet: 1 22605 0.55 MB/sec execute 169 sec latency 4159.692 ms oleg255-client.virtnet: 1 22605 0.54 MB/sec execute 170 sec latency 5159.897 ms oleg255-client.virtnet: 1 22605 0.54 MB/sec execute 171 sec latency 6160.097 ms oleg255-client.virtnet: 1 22605 0.54 MB/sec execute 172 sec latency 7160.275 ms oleg255-client.virtnet: 1 22605 0.53 MB/sec execute 173 sec latency 8160.428 ms oleg255-client.virtnet: 1 22605 0.53 MB/sec execute 174 sec latency 9160.637 ms oleg255-client.virtnet: 1 22605 0.53 MB/sec execute 175 sec latency 10160.763 ms oleg255-client.virtnet: 1 22605 0.53 MB/sec execute 176 sec latency 11160.931 ms oleg255-client.virtnet: 1 22605 0.52 MB/sec execute 177 sec latency 12161.156 ms oleg255-client.virtnet: 1 22605 0.52 MB/sec execute 178 sec latency 13161.380 ms oleg255-client.virtnet: 1 22605 0.52 MB/sec execute 179 sec latency 14161.619 ms oleg255-client.virtnet: 1 22605 0.51 MB/sec execute 180 sec latency 15161.816 ms oleg255-client.virtnet: 1 22605 0.51 MB/sec execute 181 sec latency 16162.028 ms oleg255-client.virtnet: 1 22605 0.51 MB/sec execute 182 sec latency 17162.269 ms oleg255-client.virtnet: 1 22830 0.52 MB/sec execute 183 sec latency 17638.358 ms oleg255-client.virtnet: 1 23119 0.52 MB/sec execute 184 sec latency 46.751 ms oleg255-client.virtnet: 1 23465 0.52 MB/sec execute 185 sec latency 39.883 ms oleg255-client.virtnet: 1 23775 0.52 MB/sec execute 186 sec latency 45.011 ms oleg255-client.virtnet: 1 24296 0.55 MB/sec execute 187 sec latency 53.590 ms oleg255-client.virtnet: 1 24856 0.57 MB/sec execute 188 sec latency 35.222 ms oleg255-client.virtnet: 1 25146 0.58 MB/sec execute 189 sec latency 17.359 ms oleg255-client.virtnet: 1 25364 0.58 MB/sec execute 190 sec latency 42.767 ms oleg255-client.virtnet: 1 25364 0.57 MB/sec execute 191 sec latency 1042.987 ms oleg255-client.virtnet: 1 25364 0.57 MB/sec execute 192 sec latency 2043.210 ms oleg255-client.virtnet: 1 25364 0.57 MB/sec execute 193 sec latency 3043.478 ms oleg255-client.virtnet: 1 25364 0.56 MB/sec execute 194 seFailover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:46:40 (1713484000) targets are mounted 19:46:40 (1713484000) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7460864 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:46:51 (1713484011) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:47:05 (1713484025) targets are mounted 19:47:05 (1713484025) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3696640 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7433216 1% /mnt/lustre test_70b fail mds1 12 times Failing mds1 on oleg255-server c latency 4043.764 ms oleg255-client.virtnet: 1 25364 0.56 MB/sec execute 195 sec latency 5043.995 ms oleg255-client.virtnet: 1 25364 0.56 MB/sec execute 196 sec latency 6044.203 ms oleg255-client.virtnet: 1 25364 0.56 MB/sec execute 197 sec latency 7044.532 ms oleg255-client.virtnet: 1 25364 0.55 MB/sec execute 198 sec latency 8044.780 ms oleg255-client.virtnet: 1 25364 0.55 MB/sec execute 199 sec latency 9045.017 ms oleg255-client.virtnet: 1 25364 0.55 MB/sec execute 200 sec latency 10045.248 ms oleg255-client.virtnet: 1 25364 0.54 MB/sec execute 201 sec latency 11045.402 ms oleg255-client.virtnet: 1 25364 0.54 MB/sec execute 202 sec latency 12045.535 ms oleg255-client.virtnet: 1 25364 0.54 MB/sec execute 203 sec latency 13045.717 ms oleg255-client.virtnet: 1 25364 0.54 MB/sec execute 204 sec latency 14045.985 ms oleg255-client.virtnet: 1 25364 0.53 MB/sec execute 205 sec latency 15046.269 ms oleg255-client.virtnet: 1 25364 0.53 MB/sec execute 206 sec latency 16046.533 ms oleg255-client.virtnet: 1 25364 0.53 MB/sec execute 207 sec latency 17046.775 ms oleg255-client.virtnet: 1 25468 0.53 MB/sec execute 208 sec latency 17680.297 ms oleg255-client.virtnet: 1 25690 0.53 MB/sec execute 209 sec latency 49.587 ms oleg255-client.virtnet: 1 25959 0.53 MB/sec execute 210 sec latency 44.608 ms oleg255-client.virtnet: 1 26378 0.54 MB/sec execute 211 sec latency 28.523 ms oleg255-client.virtnet: 1 26758 0.54 MB/sec execute 212 sec latency 48.528 ms oleg255-client.virtnet: 1 27273 0.55 MB/sec execute 213 sec latency 40.327 ms oleg255-client.virtnet: 1 27767 0.57 MB/sec execute 214 sec latency 41.741 ms oleg255-client.virtnet: 1 28211 0.58 MB/sec execute 215 sec latency 89.196 ms oleg255-client.virtnet: 1 28211 0.57 MB/sec execute 216 sec latency 1089.437 ms oleg255-client.virtnet: 1 28211 0.57 MB/sec execute 217 sec latency 2089.688 ms oleg255-client.virtnet: 1 28211 0.57 MB/sec execute 218 sec latency 3089.992 ms oleg255-client.virtnet: 1 28211 0.57 MB/sec execute 219 sec latency 4090.278 ms oleg255-client.virtnet: 1 28211 0.56 MB/sec execute 220 sec latency 5090.545 ms oleg255-client.virtnet: 1 28211 0.56 MB/sec execute 221 sec latency 6090.833 ms oleg255-client.virtnet: 1 28211 0.56 MB/sec execute 222 sec latency 7091.137 ms oleg255-client.virtnet: 1 28211 0.56 MB/sec execute 223 sec latency 8091.408 ms oleg255-client.virtnet: 1 28211 0.55 MB/sec execute 224 sec latency 9091.622 ms oleg255-client.virtnet: 1 28211 0.55 MB/sec execute 225 sec latency 10091.839 ms oleg255-client.virtnet: 1 28211 0.55 MB/sec execute 226 sec latency 11092.038 ms oleg255-client.virtnet: 1 28211 0.55 MB/sec execute 227 sec latency 12092.237 ms oleg255-client.virtnet: 1 28211 0.54 MB/sec execute 228 sec latency 13092.409 ms oleg255-client.virtnet: 1 28211 0.54 MB/sec execute 229 sec latency 14092.621 ms oleg255-client.virtnet: 1 28211 0.54 MB/sec execute 230 sec latency 15092.856 ms oleg255-client.virtnet: 1 28211 0.54 MB/sec execute 231 sec latency 16092.990 ms oleg255-client.virtnet: 1 28695 0.55 MB/sec execute 232 sec latency 16102.304 ms oleg255-client.virtnet: 1 28974 0.55 MB/sec execute 233 sec latency 15.917 ms oleg255-client.virtnet: 1 29209 0.55 MB/sec execute 234 sec latency 71.208 ms oleg255-client.virtnet: 1 29424 0.55 MB/sec execute 235 sec latency 46.427 ms oleg255-client.virtnet: 1 29852 0.57 MB/sec execute 236 sec latency 16.168 ms oleg255-client.virtnet: 1 30119 0.56 MB/sec execute 237 sec latency 38.941 ms oleg255-client.virtnet: 1 30444 0.56 MB/sec execute 238 sec latency Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:47:15 (1713484035) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:47:28 (1713484048) targets are mounted 19:47:28 (1713484048) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 42.878 ms oleg255-client.virtnet: 1 30619 0.56 MB/sec execute 239 sec latency 562.771 ms oleg255-client.virtnet: 1 30619 0.56 MB/sec execute 240 sec latency 1562.950 ms oleg255-client.virtnet: 1 30619 0.56 MB/sec execute 241 sec latency 2563.191 ms oleg255-client.virtnet: 1 30619 0.55 MB/sec execute 242 sec latency 3563.411 ms oleg255-client.virtnet: 1 30619 0.55 MB/sec execute 243 sec latency 4563.635 ms oleg255-client.virtnet: 1 30619 0.55 MB/sec execute 244 sec latency 5563.911 ms oleg255-client.virtnet: 1 30619 0.55 MB/sec execute 245 sec latency 6564.154 ms oleg255-client.virtnet: 1 30619 0.55 MB/sec execute 246 sec latency 7564.367 ms oleg255-client.virtnet: 1 30619 0.54 MB/sec execute 247 sec latency 8564.614 ms oleg255-client.virtnet: 1 30619 0.54 MB/sec execute 248 sec latency 9564.804 ms oleg255-client.virtnet: 1 30619 0.54 MB/sec execute 249 sec latency 10564.983 ms oleg255-client.virtnet: 1 30619 0.54 MB/sec execute 250 sec latency 11565.254 ms oleg255-client.virtnet: 1 30619 0.54 MB/sec execute 251 sec latency 12565.488 ms oleg255-client.virtnet: 1 30619 0.53 MB/sec execute 252 sec latency 13565.740 ms oleg255-client.virtnet: 1 30619 0.53 MB/sec execute 253 sec latency 14565.937 ms oleg255-client.virtnet: 1 30619 0.53 MB/sec execute 254 sec latency 15566.121 ms oleg255-client.virtnet: 1 30619 0.53 MB/sec execute 255 sec latency 16566.294 ms oleg255-client.virtnet: 1 30619 0.52 MB/sec execute 256 sec latency 17566.450 ms oleg255-client.virtnet: 1 30619 0.52 MB/sec execute 257 sec latency 18566.636 ms oleg255-client.virtnet: 1 30808 0.53 MB/sec execute 258 sec latency 19021.276 ms oleg255-client.virtnet: 1 31295 0.54 MB/sec execute 259 sec latency 42.607 ms oleg255-client.virtnet: 1 31776 0.55 MB/sec execute 260 sec latency 41.642 ms oleg255-client.virtnet: 1 32179 0.56 MB/sec execute 261 sec latency 15.562 ms oleg255-client.virtnet: 1 32401 0.56 MB/sec execute 262 sec latency 16.253 ms oleg255-client.virtnet: 1 32697 0.56 MB/sec execute 263 sec latency 41.955 ms oleg255-client.virtnet: 1 32891 0.56 MB/sec execute 264 sec latency 53.559 ms oleg255-client.virtnet: 1 33215 0.57 MB/sec execute 265 sec latency 40.696 ms oleg255-client.virtnet: 1 33574 0.57 MB/sec execute 266 sec latency 42.385 ms oleg255-client.virtnet: 1 33856 0.57 MB/sec execute 267 sec latency 48.028 ms oleg255-client.virtnet: 1 34292 0.58 MB/sec execute 268 sec latency 23.298 ms oleg255-client.virtnet: 1 34568 0.58 MB/sec execute 269 sec latency 40.856 ms oleg255-client.virtnet: 1 35174 0.60 MB/sec execute 270 sec latency 44.441 ms oleg255-client.virtnet: 1 35637 0.61 MB/sec execute 271 sec latency 27.982 ms oleg255-client.virtnet: 1 35906 0.61 MB/sec execute 272 sec latency 15.769 ms oleg255-client.virtnet: 1 36196 0.61 MB/sec execute 273 sec latency 48.381 ms oleg255-client.virtnet: 1 36442 0.61 MB/sec execute 274 sec latency 44.624 ms oleg255-client.virtnet: 1 36766 0.61 MB/sec execute 275 sec latency 45.756 ms oleg255-client.virtnet: 1 37146 0.62 MB/sec execute 276 sec latency 40.270 ms oleg255-client.virtnet: 1 37446 0.62 MB/sec execute 277 sec latency 43.736 ms oleg255-client.virtnet: 1 37857 0.62 MB/sec execute 278 sec latency 45.980 ms oleg255-client.virtnet: 1 38266 0.64 MB/sec execute 279 sec latency 41.700 ms oleg255-client.virtnet: 1 38809 0.65 MB/sec execute 280 sec latency 39.589 ms oleg255-client.virtnet: 1 39224 0.66 MB/sec execute 281 sec latency 25.034 ms oleg255-client.virtnet: 1 39482 0.66 MB/sec execute 282 sec latency 16.217 ms oleg255-client.virtnet: 1 39778 0.66 MB/sec execute 283 sec latency 60.301 ms oleg255-client.virtnet: 1 39984 0.66 MB/sec execute 284 sec latency 48.693 ms oleg255-client.virtnet: 1 40308 0.66 MB/sec execute 285 sec latency 37.296 ms oleg255-client.virtnet: 1 40694 0.67 MB/sec execute 286 sec latency 40.021 ms oleg255-client.virtnet: 1 41009 0.67 MB/sec execute 287 sec latency 48.116 ms oleg255-client.virtnet: 1 41436 0.67 MB/sec execute 288 sec latency 44.729 ms oleg255-client.virtnet: 1 41839 0.68 MB/sec execute 289 sec latency 42.497 ms oleg255-client.virtnet: 1 42406 0.69 MB/sec execute 290 sec latency 40.924 ms oleg255-client.virtnet: 1 42822 0.70 MB/sec execute 291 sec latency 17.053 ms oleg255-client.virtnet: 1 43049 0.70 MB/sec execute 292 sec latency 15.996 ms oleg255-client.virtnet: 1 43332 0.70 MB/sec execute 293 sec latency 72.700 ms oleg255-client.virtnet: 1 43701 0.70 MB/sec execute 294 sec latency 26.478 ms oleg255-client.virtnet: 1 44091 0.71 MB/sec execute 295 sec latency 38.915 ms oleg255-client.virtnet: 1 44400 0.71 MB/sec execute 296 sec latency 49.640 ms oleg255-client.virtnet: 1 44944 0.71 MB/sec execute 297 sec latency 19.465 ms oleg255-client.virtnet: 1 45348 0.72 MB/sec execute 298 sec latency 43.072 ms oleg255-client.virtnet: 1 45901 0.73 MB/sec execute 299 sec latency 38.696 ms oleg255-client.virtnet: 1 cleanup 300 sec oleg255-client.virtnet: 0 cleanup 300 sec oleg255-client.virtnet: oleg255-client.virtnet: Operation Count AvgLat MaxLat oleg255-client.virtnet: ---------------------------------------- oleg255-client.virtnet: NTCreateX 7009 22.338 27427.077 oleg255-client.virtnet: Close 5176 11.966 17577.762 oleg255-client.virtnet: Rename 301 11.883 17.680 oleg255-client.virtnet: Unlink 1400 3.199 8.641 oleg255-client.virtnet: Qpathinfo 6398 7.556 17418.646 oleg255-client.virtnet: Qfileinfo 1128 0.202 3.057 oleg255-client.virtnet: Qfsinfo 1179 0.191 6.652 oleg255-client.virtnet: Sfileinfo 566 5.986 8.979 oleg255-client.virtnet: Find 2492 0.696 30.121 oleg255-client.virtnet: WriteX 3508 1.234 7.151 oleg255-client.virtnet: ReadX 11257 0.026 2.361 oleg255-client.virtnet: LockX 24 1.906 2.144 oleg255-client.virtnet: UnlockX 24 1.971 2.263 oleg255-client.virtnet: Flush 489 29.619 73.619 oleg255-client.virtnet: oleg255-client.virtnet: Throughput 0.733656 MB/sec 1 clients 1 procs max_latency=27427.103 ms oleg255-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg255-client.virtnet at Thu Apr 18 19:48:16 EDT 2024 with return code 0 oleg255-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg255-client.virtnet oleg255-client.virtnet: /mnt/lustre/d70b.replay-single/oleg255-client.virtnet /mnt/lustre/d70b.replay-single/oleg255-client.virtnet oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg255-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg255-client.virtnet: removed directory: 'clients/client0' oleg255-client.virtnet: removed directory: 'clients' oleg255-client.virtnet: removed 'client.txt' oleg255-client.virtnet: /mnt/lustre/d70b.replay-single/oleg255-client.virtnet oleg255-client.virtnet: dbench successfully finished PASS 70b (364s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70c: tar 1mdts recovery ============ 19:48:20 (1713484100) Starting client oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar 27400 tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 11904 2196608 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3231744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3224576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 6456320 1% /mnt/lustre test_70c fail mds1 1 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:50:34 (1713484234) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:50:47 (1713484247) targets are mounted 19:50:47 (1713484247) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 11904 2196608 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3190784 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3168256 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 6359040 1% /mnt/lustre test_70c fail mds1 2 times Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:53:13 (1713484393) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:53:27 (1713484407) targets are mounted 19:53:27 (1713484407) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (369s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 19:54:31 (1713484471) SKIP: replay-single test_70d needs >= 2 MDTs SKIP 70d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70e: rename cross-MDT with random fails ========================================================== 19:54:34 (1713484474) SKIP: replay-single test_70e needs >= 2 MDTs SKIP 70e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 19:54:38 (1713484478) mount clients oleg255-client.virtnet ... Starting client oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3747840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 14336 7501824 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:54:46 (1713484486) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... Started lustre-OST0000 19:55:00 (1713484500) targets are mounted 19:55:00 (1713484500) facet_failover done ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 9216 3749888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7502848 1% /mnt/lustre ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... test_70f failing OST 2 times Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:55:13 (1713484513) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Started lustre-OST0000 19:55:27 (1713484527) targets are mounted 19:55:27 (1713484527) facet_failover done Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 14336 7497728 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... test_70f failing OST 3 times Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:55:41 (1713484541) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... Started lustre-OST0000 19:55:55 (1713484555) targets are mounted 19:55:55 (1713484555) facet_failover done ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 9216 3747840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7501824 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... test_70f failing OST 4 times Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:56:08 (1713484568) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Started lustre-OST0000 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... 19:56:22 (1713484582) targets are mounted 19:56:22 (1713484582) facet_failover done ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 9216 3749888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7502848 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... test_70f failing OST 5 times Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 19:56:36 (1713484596) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Started lustre-OST0000 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... 19:56:50 (1713484610) targets are mounted 19:56:50 (1713484610) facet_failover done ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b3c8d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b3c8d000.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg255-client.virtnet' ... PASS 70f (140s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 19:57:00 (1713484620) SKIP: replay-single test_71a needs >= 2 MDTs SKIP 71a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 19:57:03 (1713484623) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:57:07 (1713484627) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:57:21 (1713484641) targets are mounted 19:57:21 (1713484641) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (39s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 19:57:44 (1713484664) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6896 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:57:48 (1713484668) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:58:02 (1713484682) targets are mounted 19:58:02 (1713484682) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (39s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 19:58:26 (1713484706) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg255-server Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:58:29 (1713484709) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 19:58:43 (1713484723) targets are mounted 19:58:43 (1713484723) facet_failover done Starting client oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 19:58:53 (1713484733) SKIP: replay-single test_80a needs >= 2 MDTs SKIP 80a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 19:58:57 (1713484737) SKIP: replay-single test_80b needs >= 2 MDTs SKIP 80b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 19:59:00 (1713484740) SKIP: replay-single test_80c needs >= 2 MDTs SKIP 80c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 19:59:03 (1713484743) SKIP: replay-single test_80d needs >= 2 MDTs SKIP 80d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 19:59:07 (1713484747) SKIP: replay-single test_80e needs >= 2 MDTs SKIP 80e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 19:59:10 (1713484750) SKIP: replay-single test_80f needs >= 2 MDTs SKIP 80f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 19:59:13 (1713484753) SKIP: replay-single test_80g needs >= 2 MDTs SKIP 80g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 19:59:17 (1713484757) SKIP: replay-single test_80h needs >= 2 MDTs SKIP 80h (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 19:59:20 (1713484760) SKIP: replay-single test_81a needs >= 2 MDTs SKIP 81a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 19:59:23 (1713484763) SKIP: replay-single test_81b needs >= 2 MDTs SKIP 81b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 19:59:27 (1713484767) SKIP: replay-single test_81c needs >= 2 MDTs SKIP 81c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 19:59:30 (1713484770) SKIP: replay-single test_81d needs >= 2 MDTs SKIP 81d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 19:59:33 (1713484773) SKIP: replay-single test_81e needs >= 2 MDTs SKIP 81e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 19:59:37 (1713484777) SKIP: replay-single test_81f needs >= 2 MDTs SKIP 81f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 19:59:40 (1713484780) SKIP: replay-single test_81g needs >= 2 MDTs SKIP 81g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 19:59:43 (1713484783) SKIP: replay-single test_81h needs >= 2 MDTs SKIP 81h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 84a: stale open during export disconnect ========================================================== 19:59:47 (1713484787) fail_loc=0x80000144 total: 1 open/close in 0.01 seconds: 133.98 ops/second PASS 84a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 19:59:53 (1713484793) before recovery: unused locks count = 201 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 19:59:57 (1713484797) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:00:10 (1713484810) targets are mounted 20:00:10 (1713484810) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 20:00:18 (1713484818) before recovery: unused locks count = 100 Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 20:00:25 (1713484825) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 20:00:39 (1713484839) targets are mounted 20:00:39 (1713484839) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 20:00:47 (1713484847) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) PASS 86 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 87a: write replay ================== 20:00:56 (1713484856) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3755008 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 18432 7520256 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.173746 s, 48.3 MB/s Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 20:01:00 (1713484860) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 20:01:14 (1713484874) targets are mounted 20:01:14 (1713484874) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0666738 s, 126 MB/s PASS 87a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 20:01:22 (1713484882) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3757056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7522304 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.160819 s, 52.2 MB/s 8+0 records in 8+0 records out 8 bytes (8 B) copied, 0.00285202 s, 2.8 kB/s Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 20:01:27 (1713484887) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 20:01:41 (1713484901) targets are mounted 20:01:41 (1713484901) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes (72 B) copied, 0.00547494 s, 13.2 kB/s PASS 87b (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 88: MDS should not assign same objid to different files ========================================================== 20:01:49 (1713484909) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3756032 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 17408 7521280 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3756032 1% /mnt/lustre[OST:0] R lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 17408 7521280 1% /mnt/lustre before test: last_id = 10817, next_id = 10788 Creating to objid 10817 on ost lustre-OST0000... total: 31 open/close in 0.16 seconds: 194.34 ops/second total: 8 open/close in 0.04 seconds: 205.11 ops/second before recovery: last_id = 10913, next_id = 10826 Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server Failover mds1 to oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 10857, next_id = 10826 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0437277 s, 12.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.022789 s, 23.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0399877 s, 13.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0409238 s, 12.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0423141 s, 12.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0191655 s, 27.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0193034 s, 27.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0190455 s, 27.5 MB/s -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10788 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10789 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10790 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10791 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10792 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10793 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10794 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10795 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10796 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10797 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10798 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10799 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10800 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10801 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10802 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10803 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10804 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10805 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10806 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10807 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10808 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10809 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10810 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10811 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10812 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10813 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10814 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10815 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10816 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10817 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10818 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10819 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10820 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10821 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10822 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10823 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10824 -rw-r--r-- 1 root root 0 Apr 18 20:01 /mnt/lustre/d88.replay-single/f-10825 -rw-r--r-- 1 root root 524288 Apr 18 20:02 /mnt/lustre/d88.replay-single/f-10829 -rw-r--r-- 1 root root 524288 Apr 18 20:02 /mnt/lustre/d88.replay-single/f-10830 -rw-r--r-- 1 root root 524288 Apr 18 20:02 /mnt/lustre/d88.replay-single/f-10831 -rw-r--r-- 1 root root 524288 Apr 18 20:02 /mnt/lustre/d88.replay-single/f-10832 -rw-r--r-- 1 root root 524288 Apr 18 20:02 /mnt/lustre/d88.replay-single/f-10833 -rw-r--r-- 1 root root 524288 Apr 18 20:02 /mnt/lustre/d88.replay-single/f-10834 -rw-r--r-- 1 root root 524288 Apr 18 20:02 /mnt/lustre/d88.replay-single/f-10835 -rw-r--r-- 1 root root 524288 Apr 18 20:02 /mnt/lustre/d88.replay-single/f-10836 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0173818 s, 30.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0178745 s, 29.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0170443 s, 30.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0168823 s, 31.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.01623 s, 32.3 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0158701 s, 33.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0163487 s, 32.1 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0401409 s, 13.1 MB/s PASS 88 (52s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 89: no disk space leak on late ost connection ========================================================== 20:02:43 (1713484963) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg255-server mds-ost sync done. sleep 5 for ZFS zfs Waiting for MDT destroys to complete 10+0 records in 10+0 records out 41943040 bytes (42 MB) copied, 0.210935 s, 199 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg255-server Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:02:55 (1713484975) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:03:09 (1713484989) targets are mounted 20:03:09 (1713484989) facet_failover done Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg255-server mds-ost sync done. sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete free_before: 7517184 free_after: 7517184 PASS 89 (122s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 20:04:47 (1713485087) Create the files Fail ost2 lustre-OST0001_UUID, display the list of affected files Stopping /mnt/lustre-ost2 (opts:) on oleg255-server General Query: lfs find /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f1 /mnt/lustre/d90.replay-single/f0 Querying files on shutdown ost2: lfs find --obd lustre-OST0001_UUID /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f1 Check getstripe: /home/green/git/lustre-release/lustre/utils/lfs getstripe -r --obd lustre-OST0001_UUID /mnt/lustre/d90.replay-single/all lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 10690 0x29c2 0x280000400 * /mnt/lustre/d90.replay-single/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 10691 0x29c3 0x280000400 * /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f1 Failover ost2 to oleg255-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0001 PASS 90 (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 93a: replay + reconnect ============ 20:05:10 (1713485110) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00285228 s, 359 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server 20:05:13 (1713485113) shut down Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 20:05:27 (1713485127) targets are mounted 20:05:27 (1713485127) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (60s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 93b: replay + reconnect on mds ===== 20:06:12 (1713485172) total: 20 open/close in 0.10 seconds: 203.97 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:06:15 (1713485175) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:06:29 (1713485189) targets are mounted 20:06:29 (1713485189) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (101s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 20:07:55 (1713485275) SKIP: replay-single test_100a needs >= 2 MDTs SKIP 100a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 20:07:57 (1713485277) SKIP: replay-single test_100b needs >= 2 MDTs SKIP 100b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 20:08:00 (1713485280) SKIP: replay-single test_100c needs >= 2 MDTs SKIP 100c (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 20:08:02 (1713485282) SKIP: replay-single test_100d needs > 1 MDTs SKIP 100d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 20:08:05 (1713485285) SKIP: replay-single test_100e needs >= 2 MDTs SKIP 100e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 20:08:07 (1713485287) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3747840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 25600 7513088 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Failover mds1 to oleg255-server Starting mds1: -o localrecov -o abort_recovery lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 101 (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 20:08:36 (1713485316) creating 7 files ... fail_loc=0x159 launch 7 chmod in parallel (20:08:37) ... fail_loc=0 done (20:08:53) /mnt/lustre/d102a.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-7 has perms 0600 OK PASS 102a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 20:08:56 (1713485336) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (20:08:56) ... fail_loc=0 done (20:09:12) /mnt/lustre/d102b.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-7 has perms 0600 OK PASS 102b (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 20:09:15 (1713485355) creating 7 files ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3344384 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3350528 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 25600 6694912 1% /mnt/lustre fail_loc=0x15a launch 7 chmod in parallel (20:09:17) ... fail_loc=0 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:09:20 (1713485360) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:09:32 (1713485372) targets are mounted 20:09:32 (1713485372) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec done (20:09:36) /mnt/lustre/d102c.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-7 has perms 0600 OK PASS 102c (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight ========================================================== 20:09:38 (1713485378) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (20:09:39) ... fail_loc=0 Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:09:42 (1713485382) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:09:54 (1713485394) targets are mounted 20:09:54 (1713485394) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec done (20:09:58) /mnt/lustre/d102d.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-7 has perms 0600 OK PASS 102d (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 103: Check otr_next_id overflow ==== 20:10:02 (1713485402) fail_loc=0x80000162 total: 30 open/close in 0.08 seconds: 358.90 ops/second Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:10:04 (1713485404) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:10:17 (1713485417) targets are mounted 20:10:17 (1713485417) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 103 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 20:10:23 (1713485423) SKIP: replay-single test_110a needs >= 2 MDTs SKIP 110a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 20:10:26 (1713485426) SKIP: replay-single test_110b needs >= 2 MDTs SKIP 110b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 20:10:28 (1713485428) SKIP: replay-single test_110c needs >= 2 MDTs SKIP 110c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 20:10:31 (1713485431) SKIP: replay-single test_110d needs >= 2 MDTs SKIP 110d (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 20:10:33 (1713485433) SKIP: replay-single test_110e needs >= 2 MDTs SKIP 110e (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-single test_110f skipping excluded test 110f debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 20:10:36 (1713485436) SKIP: replay-single test_110g needs >= 2 MDTs SKIP 110g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 20:10:38 (1713485438) SKIP: replay-single test_111a needs >= 2 MDTs SKIP 111a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 20:10:41 (1713485441) SKIP: replay-single test_111b needs >= 2 MDTs SKIP 111b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 20:10:43 (1713485443) SKIP: replay-single test_111c needs >= 2 MDTs SKIP 111c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 20:10:46 (1713485446) SKIP: replay-single test_111d needs >= 2 MDTs SKIP 111d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 20:10:48 (1713485448) SKIP: replay-single test_111e needs >= 2 MDTs SKIP 111e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 20:10:51 (1713485451) SKIP: replay-single test_111f needs >= 2 MDTs SKIP 111f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 20:10:53 (1713485453) SKIP: replay-single test_111g needs >= 2 MDTs SKIP 111g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 20:10:56 (1713485456) SKIP: replay-single test_112a needs >= 4 MDTs SKIP 112a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 20:10:58 (1713485458) SKIP: replay-single test_112b needs >= 4 MDTs SKIP 112b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 20:11:01 (1713485461) SKIP: replay-single test_112c needs >= 4 MDTs SKIP 112c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 20:11:03 (1713485463) SKIP: replay-single test_112d needs >= 4 MDTs SKIP 112d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 20:11:06 (1713485466) SKIP: replay-single test_112e needs >= 4 MDTs SKIP 112e (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 20:11:08 (1713485468) SKIP: replay-single test_112f needs >= 4 MDTs SKIP 112f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 20:11:10 (1713485470) SKIP: replay-single test_112g needs >= 4 MDTs SKIP 112g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 20:11:13 (1713485473) SKIP: replay-single test_112h needs >= 4 MDTs SKIP 112h (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 20:11:15 (1713485475) SKIP: replay-single test_112i needs >= 4 MDTs SKIP 112i (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 20:11:17 (1713485477) SKIP: replay-single test_112j needs >= 4 MDTs SKIP 112j (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 20:11:20 (1713485480) SKIP: replay-single test_112k needs >= 4 MDTs SKIP 112k (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 20:11:22 (1713485482) SKIP: replay-single test_112l needs >= 4 MDTs SKIP 112l (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 20:11:24 (1713485484) SKIP: replay-single test_112m needs >= 4 MDTs SKIP 112m (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 20:11:27 (1713485487) SKIP: replay-single test_112n needs >= 4 MDTs SKIP 112n (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 115: failover for create/unlink striped directory ========================================================== 20:11:29 (1713485489) SKIP: replay-single test_115 needs >= 2 MDTs SKIP 115 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 116a: large update log master MDT recovery ========================================================== 20:11:32 (1713485492) SKIP: replay-single test_116a needs >= 2 MDTs SKIP 116a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 116b: large update log slave MDT recovery ========================================================== 20:11:34 (1713485494) SKIP: replay-single test_116b needs >= 2 MDTs SKIP 116b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 20:11:36 (1713485496) SKIP: replay-single test_117 needs >= 4 MDTs SKIP 117 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 20:11:39 (1713485499) SKIP: replay-single test_118 needs >= 2 MDTs SKIP 118 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 20:11:41 (1713485501) SKIP: replay-single test_119 needs >= 2 MDTs SKIP 119 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 20:11:44 (1713485504) SKIP: replay-single test_120 needs >= 2 MDTs SKIP 120 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 121: lock replay timed out and race ========================================================== 20:11:47 (1713485507) multiop /mnt/lustre/f121.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.6896 Stopping /mnt/lustre-mds1 (opts:) on oleg255-server Failover mds1 to oleg255-server fail_loc=0x721 fail_val=0 at_max=0 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0x0 at_max=600 PASS 121 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 20:12:14 (1713485534) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3344384 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 25600 7109632 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:12:16 (1713485536) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:12:29 (1713485549) targets are mounted 20:12:29 (1713485549) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 130b: DoM file create (inherited) replay ========================================================== 20:12:34 (1713485554) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3344384 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 25600 7109632 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:12:37 (1713485557) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:12:49 (1713485569) targets are mounted 20:12:49 (1713485569) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 131a: DoM file write lock replay === 20:12:55 (1713485575) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3344384 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 25600 7109632 1% /mnt/lustre 1+0 records in 1+0 records out 8 bytes (8 B) copied, 0.00116804 s, 6.8 kB/s Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:12:57 (1713485577) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:13:10 (1713485590) targets are mounted 20:13:10 (1713485590) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 131a (19s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-single test_131b skipping excluded test 131b debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 132a: PFL new component instantiate replay ========================================================== 20:13:16 (1713485596) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3344384 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 25600 7109632 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145259 s, 72.2 MB/s /mnt/lustre/f132a.replay-single lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8a:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8b:0x0] } Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:13:18 (1713485598) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:13:31 (1713485611) targets are mounted 20:13:31 (1713485611) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f132a.replay-single lcm_layout_gen: 1 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8a:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8b:0x0] } PASS 132a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 20:13:37 (1713485617) SKIP: replay-single test_133 needs >= 2 MDTs SKIP 133 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 134: replay creation of a file created in a pool ========================================================== 20:13:39 (1713485619) Creating new pool oleg255-server: Pool lustre.pool_134 created Adding targets to pool oleg255-server: OST lustre-OST0001_UUID added to pool lustre.pool_134 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3747840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 26624 7512064 1% /mnt/lustre Failing mds1 on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server 20:13:46 (1713485626) shut down Failover mds1 to oleg255-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 20:13:58 (1713485638) targets are mounted 20:13:58 (1713485638) facet_failover done oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Destroy the created pools: pool_134 lustre.pool_134 oleg255-server: OST lustre-OST0001_UUID removed from pool lustre.pool_134 oleg255-server: Pool lustre.pool_134 destroyed Waiting 90s for 'foo' PASS 134 (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 135: Server failure in lock replay phase ========================================================== 20:14:10 (1713485650) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3747840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 26624 7512064 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 Stopping /mnt/lustre-ost1 (opts:) on oleg255-server Failover ost1 to oleg255-server oleg255-server: oleg255-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg255-server Failover ost1 to oleg255-server oleg255-server: oleg255-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server End of sync Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0001 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 5 ldlm.cancel_unused_locks_before_replay=1 PASS 135 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 20:15:13 (1713485713) SKIP: replay-single test_136 needs > 2 MDTs SKIP 136 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 20:15:15 (1713485715) SKIP: replay-single test_200 Need remote client SKIP 200 (1s) debug_raw_pointers=0 debug_raw_pointers=0 == replay-single test complete, duration 5061 sec ======== 20:15:16 (1713485716) === replay-single: start cleanup 20:15:17 (1713485717) === === replay-single: finish cleanup 20:15:19 (1713485719) ===