-----============= acceptance-small: replay-ost-single ============----- Fri Apr 19 08:35:15 EDT 2024 excepting tests: === replay-ost-single: start setup 08:35:16 (1713530116) === oleg359-client.virtnet: executing check_config_client /mnt/lustre oleg359-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg359-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012ab19800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012ab19800.idle_timeout=debug disable quota as required oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-ost-single: finish setup 08:35:25 (1713530125) === /mnt/lustre/d0.replay-ost-single stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: 0 osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=1 osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=1 Creating to objid 33 on ost lustre-OST0001... Creating to objid 33 on ost lustre-OST0001... Creating to objid 33 on ost lustre-OST0000... Creating to objid 33 on ost lustre-OST0000... total: 33 open/close in 0.10 seconds: 328.97 ops/second total: 33 open/close in 0.10 seconds: 323.26 ops/second total: 33 open/close in 0.11 seconds: 295.64 ops/second total: 33 open/close in 0.11 seconds: 291.11 ops/second osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=0 osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=0 osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=0 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 0a: target handle mismatch (bug 5317) ========================================================== 08:35:37 (1713530137) Stopping client oleg359-client.virtnet /mnt/lustre (opts:-f) fail_loc=0x80000211 Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 3833116 1796 3605224 1% /mnt/lustre PASS 0a (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 0b: empty replay =============== 08:35:51 (1713530151) Failing ost1 on oleg359-server Stopping /mnt/lustre-ost1 (opts:) on oleg359-server 08:35:52 (1713530152) shut down Failover ost1 to oleg359-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 08:36:05 (1713530165) targets are mounted 08:36:05 (1713530165) facet_failover done oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 0b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 1: touch ======================= 08:36:11 (1713530171) Failing ost1 on oleg359-server Stopping /mnt/lustre-ost1 (opts:) on oleg359-server 08:36:13 (1713530173) shut down Failover ost1 to oleg359-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 08:36:26 (1713530186) targets are mounted 08:36:26 (1713530186) facet_failover done oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /mnt/lustre/d0.replay-ost-single/f1.replay-ost-single has type file OK PASS 1 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 2: |x| 10 open(O_CREAT)s ======= 08:36:32 (1713530192) Failing ost1 on oleg359-server Stopping /mnt/lustre-ost1 (opts:) on oleg359-server 08:36:33 (1713530193) shut down Failover ost1 to oleg359-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 08:36:46 (1713530206) targets are mounted 08:36:46 (1713530206) facet_failover done oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 2 (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 3: Fail OST during write, with verification ========================================================== 08:36:52 (1713530212) Failing ost1 on oleg359-server 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.141094 s, 37.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg359-server 08:36:53 (1713530213) shut down Failover ost1 to oleg359-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 08:37:06 (1713530226) targets are mounted 08:37:06 (1713530226) facet_failover done oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 3 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y