== recovery-small test 51: failover MDS during recovery == 05:09:43 (1713431383)
fail_loc=0x00001310
Failing mds1 on oleg404-server
Stopping /mnt/lustre-mds1 (opts:) on oleg404-server
05:09:46 (1713431386) shut down
Failover mds1 to oleg404-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all
pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1
Started lustre-MDT0000
05:09:59 (1713431399) targets are mounted
05:09:59 (1713431399) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg404-server
Stopping /mnt/lustre-mds1 (opts:) on oleg404-server
05:10:02 (1713431402) shut down
Failover mds1 to oleg404-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all
pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1
Started lustre-MDT0000
05:10:14 (1713431414) targets are mounted
05:10:14 (1713431414) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg404-server
Stopping /mnt/lustre-mds1 (opts:) on oleg404-server
05:10:21 (1713431421) shut down
Failover mds1 to oleg404-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all
pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1
Started lustre-MDT0000
05:10:33 (1713431433) targets are mounted
05:10:33 (1713431433) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg404-server
Stopping /mnt/lustre-mds1 (opts:) on oleg404-server
05:10:45 (1713431445) shut down
Failover mds1 to oleg404-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all
pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1
Started lustre-MDT0000
05:10:57 (1713431457) targets are mounted
05:10:57 (1713431457) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg404-server
Stopping /mnt/lustre-mds1 (opts:) on oleg404-server
05:11:19 (1713431479) shut down
Failover mds1 to oleg404-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all
pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1
Started lustre-MDT0000
05:11:31 (1713431491) targets are mounted
05:11:31 (1713431491) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg404-server
Stopping /mnt/lustre-mds1 (opts:) on oleg404-server
05:11:58 (1713431518) shut down
Failover mds1 to oleg404-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all
pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1
Started lustre-MDT0000
05:12:10 (1713431530) targets are mounted
05:12:10 (1713431530) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg404-server
Stopping /mnt/lustre-mds1 (opts:) on oleg404-server
05:12:42 (1713431562) shut down
Failover mds1 to oleg404-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all
pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1
Started lustre-MDT0000
05:12:55 (1713431575) targets are mounted
05:12:55 (1713431575) facet_failover done
writemany returned 0