-----============= acceptance-small: replay-dual ============----- Fri Apr 19 08:34:58 EDT 2024 excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 08:35:00 (1713530100) === Starting client oleg118-client.virtnet: -o user_xattr,flock oleg118-server@tcp:/lustre /mnt/lustre2 Started clients oleg118-client.virtnet: 192.168.201.118@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg118-client.virtnet: executing check_config_client /mnt/lustre oleg118-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg118-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b599c800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b5a61800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b599c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5a61800.idle_timeout=debug disable quota as required oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 08:35:08 (1713530108) === osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=1 Creating to objid 33 on ost lustre-OST0000... Creating to objid 33 on ost lustre-OST0001... total: 33 open/close in 0.47 seconds: 70.36 ops/second total: 33 open/close in 0.48 seconds: 69.24 ops/second osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=0 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 0a: expired recovery with lost client ========================================================== 08:35:21 (1713530121) Check file is LU482_FAILED=/tmp/replay-dual.lu482.j6JEHM UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 50 open/close in 0.33 seconds: 153.75 ops/second fail_loc=0x80000514 Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 08:35:24 (1713530124) shut down Failover mds1 to oleg118-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 08:35:37 (1713530137) targets are mounted 08:35:37 (1713530137) facet_failover done Starting client: oleg118-client.virtnet: -o user_xattr,flock oleg118-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1713530218 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second PASS 0a (98s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 0b: lost client during waiting for next transno ========================================================== 08:37:01 (1713530221)