== recovery-small test 51: failover MDS during recovery == 10:59:08 (1713365948)
fail_loc=0x00001310
Failing mds1 on oleg255-server
Stopping /mnt/lustre-mds1 (opts:) on oleg255-server
10:59:11 (1713365951) shut down
Failover mds1 to oleg255-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Started lustre-MDT0000
10:59:24 (1713365964) targets are mounted
10:59:24 (1713365964) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg255-server
Stopping /mnt/lustre-mds1 (opts:) on oleg255-server
10:59:27 (1713365967) shut down
Failover mds1 to oleg255-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Started lustre-MDT0000
10:59:40 (1713365980) targets are mounted
10:59:40 (1713365980) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg255-server
Stopping /mnt/lustre-mds1 (opts:) on oleg255-server
10:59:46 (1713365986) shut down
Failover mds1 to oleg255-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Started lustre-MDT0000
10:59:59 (1713365999) targets are mounted
10:59:59 (1713365999) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg255-server
Stopping /mnt/lustre-mds1 (opts:) on oleg255-server
11:00:11 (1713366011) shut down
Failover mds1 to oleg255-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Started lustre-MDT0000
11:00:24 (1713366024) targets are mounted
11:00:24 (1713366024) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg255-server
Stopping /mnt/lustre-mds1 (opts:) on oleg255-server
11:00:46 (1713366046) shut down
Failover mds1 to oleg255-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Started lustre-MDT0000
11:01:00 (1713366060) targets are mounted
11:01:00 (1713366060) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg255-server
Stopping /mnt/lustre-mds1 (opts:) on oleg255-server
11:01:27 (1713366087) shut down
Failover mds1 to oleg255-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Started lustre-MDT0000
11:01:40 (1713366100) targets are mounted
11:01:40 (1713366100) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg255-server
Stopping /mnt/lustre-mds1 (opts:) on oleg255-server
11:02:12 (1713366132) shut down
Failover mds1 to oleg255-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Started lustre-MDT0000
11:02:25 (1713366145) targets are mounted
11:02:25 (1713366145) facet_failover done
writemany returned 0