-----============= acceptance-small: replay-dual ============----- Fri Apr 19 08:35:14 EDT 2024
excepting tests: 14b 21b
skipping tests SLOW=no: 21b
=== replay-dual: start setup 08:35:16 (1713530116) ===
Starting client oleg316-client.virtnet:  -o user_xattr,flock oleg316-server@tcp:/lustre /mnt/lustre2
Started clients oleg316-client.virtnet: 
192.168.203.116@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
oleg316-client.virtnet: executing check_config_client /mnt/lustre
oleg316-client.virtnet: Checking config lustre mounted on /mnt/lustre
Checking servers environments
Checking clients oleg316-client.virtnet environments
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b36e9800.idle_timeout=debug
osc.lustre-OST0000-osc-ffff8800b62ec800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b36e9800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b62ec800.idle_timeout=debug
disable quota as required
oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all
osd-ldiskfs.track_declares_assert=1
=== replay-dual: finish setup 08:35:24 (1713530124) ===
osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=1
osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=1
osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=1
osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=1
Creating to objid 33 on ost lustre-OST0000...
Creating to objid 33 on ost lustre-OST0000...
Creating to objid 33 on ost lustre-OST0001...
Creating to objid 33 on ost lustre-OST0001...
total: 33 open/close in 0.20 seconds: 166.37 ops/second
total: 33 open/close in 0.20 seconds: 165.98 ops/second
total: 33 open/close in 0.20 seconds: 162.24 ops/second
total: 33 open/close in 0.21 seconds: 156.95 ops/second
osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=0
osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=0
osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=0
osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== replay-dual test 0a: expired recovery with lost client ========================================================== 08:35:37 (1713530137)
Check file is LU482_FAILED=/tmp/replay-dual.lu482.jHo4JQ
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      1414116        1840     1285848   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID      1414116        1688     1286000   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID      3833116        1796     3605224   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3833116        1796     3605224   1% /mnt/lustre[OST:1]

filesystem_summary:      7666232        3592     7210448   1% /mnt/lustre

total: 50 open/close in 0.33 seconds: 151.99 ops/second
fail_loc=0x80000514
Failing mds1 on oleg316-server
Stopping /mnt/lustre-mds1 (opts:) on oleg316-server
08:35:41 (1713530141) shut down
Failover mds1 to oleg316-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all
pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1
Started lustre-MDT0000
08:35:55 (1713530155) targets are mounted
08:35:55 (1713530155) facet_failover done