-----============= acceptance-small: replay-single ============----- Thu Apr 18 03:23:28 EDT 2024 excepting tests: 110f 131b 59 36 === replay-single: start setup 03:23:31 (1713425011) === oleg457-client.virtnet: executing check_config_client /mnt/lustre oleg457-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg457-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6d21800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6d21800.idle_timeout=debug disable quota as required oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 03:23:38 (1713425018) === debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 0a: empty replay =================== 03:23:39 (1713425019) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1772 1285916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3048 7210992 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:23:42 (1713425022) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:23:56 (1713425036) targets are mounted 03:23:56 (1713425036) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 0a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 03:24:04 (1713425044) Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 03:24:06 (1713425046) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 03:24:19 (1713425059) targets are mounted 03:24:19 (1713425059) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 20 open/close in 0.07 seconds: 283.21 ops/second - unlinked 0 (time 1713425062 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 0b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 0c: check replay-barrier =========== 03:24:25 (1713425065) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:24:28 (1713425068) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:24:42 (1713425082) targets are mounted 03:24:42 (1713425082) facet_failover done Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre rm: cannot remove '/mnt/lustre/f0c.replay-single': No such file or directory PASS 0c (89s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 0d: expired recovery with no clients ========================================================== 03:25:56 (1713425156) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:25:59 (1713425159) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:26:12 (1713425172) targets are mounted 03:26:12 (1713425172) facet_failover done Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre PASS 0d (89s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 1: simple create =================== 03:27:26 (1713425246) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:27:30 (1713425250) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:27:43 (1713425263) targets are mounted 03:27:43 (1713425263) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f1.replay-single has type file OK PASS 1 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2a: touch ========================== 03:27:52 (1713425272) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:27:55 (1713425275) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:28:09 (1713425289) targets are mounted 03:28:09 (1713425289) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2a.replay-single has type file OK PASS 2a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2b: touch ========================== 03:28:18 (1713425298) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:28:21 (1713425301) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:28:35 (1713425315) targets are mounted 03:28:35 (1713425315) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2b.replay-single has type file OK PASS 2b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2c: setstripe replay =============== 03:28:43 (1713425323) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:28:47 (1713425327) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:29:00 (1713425340) targets are mounted 03:29:00 (1713425340) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2c.replay-single has type file OK PASS 2c (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2d: setdirstripe replay ============ 03:29:09 (1713425349) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:29:12 (1713425352) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:29:26 (1713425366) targets are mounted 03:29:26 (1713425366) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d2d.replay-single has type dir OK PASS 2d (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 2e: O_CREAT|O_EXCL create replay === 03:29:34 (1713425374) fail_loc=0x8000013b UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:29:39 (1713425379) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:29:53 (1713425393) targets are mounted 03:29:53 (1713425393) facet_failover done Succeed in opening file "/mnt/lustre/f2e.replay-single"(flags=O_CREAT) oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f2e.replay-single has type file OK PASS 2e (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 03:30:01 (1713425401) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Error in opening file "/mnt/lustre/f3a.replay-single"(flags=O_DIRECTORY) 20: Not a directory Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:30:04 (1713425404) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:30:18 (1713425418) targets are mounted 03:30:18 (1713425418) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f3a.replay-single has type file OK PASS 3a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 3b: replay failed open -ENOMEM ===== 03:30:26 (1713425426) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre fail_loc=0x80000114 touch: cannot touch '/mnt/lustre/f3b.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:30:30 (1713425430) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:30:44 (1713425444) targets are mounted 03:30:44 (1713425444) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3b.replay-single: No such file or directory PASS 3b (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 3c: replay failed open -ENOMEM ===== 03:30:52 (1713425452) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre fail_loc=0x80000128 touch: cannot touch '/mnt/lustre/f3c.replay-single': Cannot allocate memory fail_loc=0 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:30:56 (1713425456) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:31:10 (1713425470) targets are mounted 03:31:10 (1713425470) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f3c.replay-single: No such file or directory PASS 3c (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 03:31:18 (1713425478) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3605480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3064 7210976 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:31:21 (1713425481) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:31:35 (1713425495) targets are mounted 03:31:35 (1713425495) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 4b: |x| rm 10 files ================ 03:31:44 (1713425504) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605320 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605336 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210656 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:31:47 (1713425507) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:32:01 (1713425521) targets are mounted 03:32:01 (1713425521) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f4b.replay-single-*: No such file or directory PASS 4b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 5: |x| 220 open(O_CREAT) =========== 03:32:10 (1713425530) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:32:15 (1713425535) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:32:29 (1713425549) targets are mounted 03:32:29 (1713425549) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (32s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 6a: mkdir + contained create ======= 03:32:43 (1713425563) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:32:46 (1713425566) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:33:00 (1713425580) targets are mounted 03:33:00 (1713425580) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d6a.replay-single has type dir OK /mnt/lustre/d6a.replay-single/f6a.replay-single has type file OK PASS 6a (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 6b: |X| rmdir ====================== 03:33:11 (1713425591) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1848 1285840 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:33:15 (1713425595) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:33:29 (1713425609) targets are mounted 03:33:29 (1713425609) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d6b.replay-single: No such file or directory PASS 6b (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 7: mkdir |X| contained create ====== 03:33:38 (1713425618) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:33:41 (1713425621) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:33:55 (1713425635) targets are mounted 03:33:55 (1713425635) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d7.replay-single has type dir OK /mnt/lustre/d7.replay-single/f7.replay-single has type file OK PASS 7 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 8: creat open |X| close ============ 03:34:04 (1713425644) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre multiop /mnt/lustre/f8.replay-single vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:34:07 (1713425647) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:34:21 (1713425661) targets are mounted 03:34:21 (1713425661) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f8.replay-single /mnt/lustre/f8.replay-single has type file OK PASS 8 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 9: |X| create (same inum/gen) ====== 03:34:29 (1713425669) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:34:32 (1713425672) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:34:46 (1713425686) targets are mounted 03:34:46 (1713425686) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old_inum == 144115305935798546, new_inum == 144115305935798546 old_inum and new_inum match PASS 9 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 10: create |X| rename unlink ======= 03:34:54 (1713425694) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:34:58 (1713425698) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:35:11 (1713425711) targets are mounted 03:35:11 (1713425711) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/f10.replay-single: No such file or directory PASS 10 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 03:35:20 (1713425720) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre new old Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:35:23 (1713425723) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:35:37 (1713425737) targets are mounted 03:35:37 (1713425737) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec new old PASS 11 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 12: open, unlink |X| close ========= 03:35:46 (1713425746) multiop /mnt/lustre/f12.replay-single vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3112 7210928 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:35:49 (1713425749) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:36:03 (1713425763) targets are mounted 03:36:03 (1713425763) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 12 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 13: open chmod 0 |x| write close === 03:36:11 (1713425771) multiop /mnt/lustre/f13.replay-single vO_wc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 /mnt/lustre/f13.replay-single has perms 00 OK UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3112 7210928 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:36:14 (1713425774) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:36:28 (1713425788) targets are mounted 03:36:28 (1713425788) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f13.replay-single has perms 00 OK /mnt/lustre/f13.replay-single has size 1 OK PASS 13 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 03:36:36 (1713425796) multiop /mnt/lustre/f14.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3112 7210928 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:36:40 (1713425800) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:36:54 (1713425814) targets are mounted 03:36:54 (1713425814) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 14 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 03:37:03 (1713425823) multiop /mnt/lustre/f15.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3112 7210928 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:37:06 (1713425826) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:37:20 (1713425840) targets are mounted 03:37:20 (1713425840) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 15 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 03:37:28 (1713425848) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3112 7210928 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:37:31 (1713425851) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:37:45 (1713425865) targets are mounted 03:37:45 (1713425865) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 16 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 03:37:54 (1713425874) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3112 7210928 1% /mnt/lustre multiop /mnt/lustre/f17.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:37:57 (1713425877) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:38:11 (1713425891) targets are mounted 03:38:11 (1713425891) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f17.replay-single has type file OK PASS 17 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 03:38:20 (1713425900) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3112 7210928 1% /mnt/lustre multiop /mnt/lustre/f18.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 pid: 24885 will close Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:38:23 (1713425903) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:38:37 (1713425917) targets are mounted 03:38:37 (1713425917) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 18 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 19: mcreate, open, write, rename === 03:38:46 (1713425926) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3112 7210928 1% /mnt/lustre old Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:38:49 (1713425929) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:39:03 (1713425943) targets are mounted 03:39:03 (1713425943) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec old PASS 19 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 03:39:12 (1713425952) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210924 1% /mnt/lustre multiop /mnt/lustre/f20a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:39:15 (1713425955) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:39:29 (1713425969) targets are mounted 03:39:29 (1713425969) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 20a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 03:39:38 (1713425978) /mnt/lustre/f20b.replay-single lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1090 0x442 0x280000401 10000+0 records in 10000+0 records out 40960000 bytes (41 MB) copied, 1.43804 s, 28.5 MB/s Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:39:42 (1713425982) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:39:56 (1713425996) targets are mounted 03:39:56 (1713425996) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg457-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for MDT destroys to complete before 3116, after 3116 PASS 20b (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 20c: check that client eviction does not affect file content ========================================================== 03:40:07 (1713426007) multiop /mnt/lustre/f20c.replay-single vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 -rw-r--r-- 1 root root 1 Apr 18 03:40 /mnt/lustre/f20c.replay-single PASS 20c (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 03:40:13 (1713426013) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1568 3605416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210888 1% /mnt/lustre multiop /mnt/lustre/f21.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:40:16 (1713426016) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:40:30 (1713426030) targets are mounted 03:40:30 (1713426030) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 21 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 03:40:40 (1713426040) multiop /mnt/lustre/f22.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605420 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210892 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:40:43 (1713426043) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:40:57 (1713426057) targets are mounted 03:40:57 (1713426057) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 03:41:05 (1713426065) multiop /mnt/lustre/f23.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:41:09 (1713426069) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:41:23 (1713426083) targets are mounted 03:41:23 (1713426083) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 03:41:31 (1713426091) multiop /mnt/lustre/f24.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:41:35 (1713426095) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:41:49 (1713426109) targets are mounted 03:41:49 (1713426109) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 24 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 03:41:58 (1713426118) multiop /mnt/lustre/f25.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:42:01 (1713426121) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:42:17 (1713426137) targets are mounted 03:42:17 (1713426137) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 25 (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 03:42:26 (1713426146) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre multiop /mnt/lustre/f26.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 multiop /mnt/lustre/f26.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:42:32 (1713426152) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:42:47 (1713426167) targets are mounted 03:42:47 (1713426167) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 26 (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 03:42:57 (1713426177) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre multiop /mnt/lustre/f27.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 multiop /mnt/lustre/f27.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:43:01 (1713426181) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:43:16 (1713426196) targets are mounted 03:43:16 (1713426196) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 27 (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 03:43:26 (1713426206) multiop /mnt/lustre/f28.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 multiop /mnt/lustre/f28.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:43:30 (1713426210) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:43:45 (1713426225) targets are mounted 03:43:45 (1713426225) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 28 (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 03:43:55 (1713426235) multiop /mnt/lustre/f29.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 multiop /mnt/lustre/f29.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:43:59 (1713426239) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:44:15 (1713426255) targets are mounted 03:44:15 (1713426255) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 29 (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 03:44:25 (1713426265) multiop /mnt/lustre/f30.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 multiop /mnt/lustre/f30.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:44:29 (1713426269) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:44:45 (1713426285) targets are mounted 03:44:45 (1713426285) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 03:44:54 (1713426294) multiop /mnt/lustre/f31.replay-single-1 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 multiop /mnt/lustre/f31.replay-single-2 vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1652 1286036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:44:59 (1713426299) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:45:14 (1713426314) targets are mounted 03:45:14 (1713426314) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 31 (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 03:45:24 (1713426324) multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 multiop /mnt/lustre/f32.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 PASS 32 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 03:45:31 (1713426331) total: 10 open/close in 0.07 seconds: 148.32 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 first stat failed: 5 total: 10 open/close in 0.07 seconds: 143.36 ops/second PASS 33a (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 33b: test fid seq allocation ======= 03:45:53 (1713426353) fail_loc=0x1311 total: 10 open/close in 0.07 seconds: 142.00 ops/second Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 first stat failed: 5 total: 10 open/close in 0.05 seconds: 200.12 ops/second PASS 33b (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 03:46:17 (1713426377) multiop /mnt/lustre/f34.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1876 1285812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1684 1286004 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 34 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 35: test recovery from llog for unlink op ========================================================== 03:46:38 (1713426398) fail_loc=0x80000119 Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 rm: cannot remove '/mnt/lustre/f35.replay-single': Input/output error oleg457-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 first stat failed: 5 Can't lstat /mnt/lustre/f35.replay-single: No such file or directory PASS 35 (17s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-single test_36 skipping ALWAYS excluded test 36 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 03:46:57 (1713426417) multiop /mnt/lustre/d37.replay-single/f37.replay-single vdD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1948 1285740 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 first stat failed: 5 PASS 37 (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 03:47:17 (1713426437) total: 800 open/close in 2.91 seconds: 274.68 ops/second - unlinked 0 (time 1713426442 ; total 0 ; last 0) total: 400 unlinks in 0 seconds: inf unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:47:26 (1713426446) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:47:40 (1713426460) targets are mounted 03:47:40 (1713426460) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1713426466 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Can't lstat /mnt/lustre/f38.replay-single-*: No such file or directory PASS 38 (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 03:47:52 (1713426472) total: 800 open/close in 2.16 seconds: 370.15 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3120 7210920 1% /mnt/lustre - unlinked 0 (time 1713426478 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:48:00 (1713426480) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:48:14 (1713426494) targets are mounted 03:48:14 (1713426494) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1713426501 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Can't lstat /mnt/lustre/f39.replay-single-*: No such file or directory PASS 39 (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 03:48:27 (1713426507) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00164588 s, 2.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0067498 s, 607 kB/s PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 42: recovery after ost failure ===== 03:48:31 (1713426511) total: 800 open/close in 2.12 seconds: 378.06 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre - unlinked 0 (time 1713426517 ; total 0 ; last 0) total: 400 unlinks in 0 seconds: inf unlinks/second debug=-1 Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 03:48:39 (1713426519) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 03:48:53 (1713426533) targets are mounted 03:48:53 (1713426533) facet_failover done wait for MDS to timeout and recover debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck - unlinked 0 (time 1713426574 ; total 0 ; last 0) total: 400 unlinks in 1 seconds: 400.000000 unlinks/second Can't lstat /mnt/lustre/f42.replay-single-*: No such file or directory PASS 42 (65s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 03:49:38 (1713426578) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2160 1285528 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre fail_loc=0x80000204 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:49:41 (1713426581) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:49:55 (1713426595) targets are mounted 03:49:55 (1713426595) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 43 (34s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 44a: race in target handle connect ========================================================== 03:50:14 (1713426614) at_max=40 1 of 10 (1713426615) service : cur 5 worst 5 (at 1713424976, 1640s ago) 4 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 2 of 10 (1713426620) service : cur 5 worst 5 (at 1713424976, 1645s ago) 5 4 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 3 of 10 (1713426626) service : cur 5 worst 5 (at 1713424976, 1651s ago) 5 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 4 of 10 (1713426631) service : cur 5 worst 5 (at 1713424976, 1656s ago) 5 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 5 of 10 (1713426637) service : cur 5 worst 5 (at 1713424976, 1662s ago) 5 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 6 of 10 (1713426642) service : cur 5 worst 5 (at 1713424976, 1667s ago) 5 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 7 of 10 (1713426648) service : cur 5 worst 5 (at 1713424976, 1673s ago) 5 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 8 of 10 (1713426653) service : cur 5 worst 5 (at 1713424976, 1678s ago) 5 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 9 of 10 (1713426659) service : cur 5 worst 5 (at 1713424976, 1684s ago) 5 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre 10 of 10 (1713426664) service : cur 5 worst 5 (at 1713424976, 1689s ago) 5 5 4 4 fail_loc=0x80000701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre fail_loc=0 at_max=600 PASS 44a (57s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 44b: race in target handle connect ========================================================== 03:51:13 (1713426673) 1 of 10 (1713426673) service : cur 5 worst 5 (at 1713424976, 1699s ago) 5 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 2 of 10 (1713426714) service : cur 40 worst 40 (at 1713426715, 0s ago) 40 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 3 of 10 (1713426734) service : cur 40 worst 40 (at 1713426715, 21s ago) 40 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 4 of 10 (1713426755) service : cur 40 worst 40 (at 1713426715, 41s ago) 40 5 4 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 5 of 10 (1713426775) service : cur 40 worst 40 (at 1713426715, 62s ago) 1 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 6 of 10 (1713426796) service : cur 40 worst 40 (at 1713426715, 82s ago) 40 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 7 of 10 (1713426816) service : cur 40 worst 40 (at 1713426715, 102s ago) 40 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 8 of 10 (1713426837) service : cur 40 worst 40 (at 1713426715, 123s ago) 40 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 9 of 10 (1713426857) service : cur 40 worst 40 (at 1713426715, 144s ago) 40 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre 10 of 10 (1713426878) service : cur 40 worst 40 (at 1713426715, 164s ago) 40 40 5 4 fail_loc=0x80000704 error: recover: Connection timed out Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7666232 3124 7210916 1% /mnt/lustre PASS 44b (226s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 44c: race in target handle connect ========================================================== 03:55:01 (1713426901) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre total: 100 create in 0.22 seconds: 450.43 ops/second fail_loc=0x80000712 Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:55:18 (1713426918) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:55:31 (1713426931) targets are mounted 03:55:31 (1713426931) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec unlink(/mnt/lustre/f44c.replay-single-0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second PASS 44c (37s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 45: Handle failed close ============ 03:55:40 (1713426940) multiop /mnt/lustre/f45.replay-single vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 /mnt/lustre/f45.replay-single has type file OK PASS 45 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 03:55:44 (1713426944) fail_loc=0x122 fail_loc=0 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:55:46 (1713426946) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:55:59 (1713426959) targets are mounted 03:55:59 (1713426959) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs path2fid: cannot get fid for 'f46.replay-single': No such file or directory PASS 46 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 03:56:08 (1713426968) total: 20 open/close in 0.07 seconds: 300.44 ops/second Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 03:56:09 (1713426969) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 03:56:23 (1713426983) targets are mounted 03:56:23 (1713426983) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0x80000204 total: 20 open/close in 0.06 seconds: 350.23 ops/second - unlinked 0 (time 1713427048 ; total 0 ; last 0) total: 20 unlinks in 0 seconds: inf unlinks/second PASS 47 (81s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 03:57:30 (1713427050) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1812 1285876 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre total: 20 open/close in 0.07 seconds: 272.63 ops/second Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:57:34 (1713427054) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:57:47 (1713427067) targets are mounted 03:57:47 (1713427067) facet_failover done fail_loc=0x80000216 total: 20 open/close in 0.07 seconds: 299.32 ops/second - unlinked 0 (time 1713427131 ; total 0 ; last 0) total: 40 unlinks in 0 seconds: inf unlinks/second PASS 48 (83s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 03:58:54 (1713427134) PASS 50 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 52: time out lock replay (3764) ==== 03:59:03 (1713427143) multiop /mnt/lustre/f52.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7497 fail_loc=0x80000157 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 03:59:05 (1713427145) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 03:59:18 (1713427158) targets are mounted 03:59:18 (1713427158) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 fail_loc=0x0 PASS 52 (80s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 04:00:25 (1713427225) fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:00:30 (1713427230) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:00:44 (1713427244) targets are mounted 04:00:44 (1713427244) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53a.replay-single-1/f has type file OK /mnt/lustre/d53a.replay-single-2/f has type file OK PASS 53a (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 04:00:52 (1713427252) multiop /mnt/lustre/d53b.replay-single-1/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7497 fail_loc=0x80000107 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:00:57 (1713427257) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:01:11 (1713427271) targets are mounted 04:01:11 (1713427271) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53b.replay-single-1/f has type file OK /mnt/lustre/d53b.replay-single-2/f has type file OK PASS 53b (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 04:01:20 (1713427280) fail_loc=0x80000107 fail_loc=0x80000115 Replay barrier on lustre-MDT0000 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:01:24 (1713427284) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:01:38 (1713427298) targets are mounted 04:01:38 (1713427298) facet_failover done fail_loc=0 /mnt/lustre/d53c.replay-single-1/f has type file OK /mnt/lustre/d53c.replay-single-2/f has type file OK PASS 53c (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 04:01:48 (1713427308) fail_loc=0x8000013b fail_loc=0 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:01:51 (1713427311) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:02:05 (1713427325) targets are mounted 04:02:05 (1713427325) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53d.replay-single-1/f has type file OK /mnt/lustre/d53d.replay-single-2/f has type file OK PASS 53d (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 04:02:14 (1713427334) fail_loc=0x119 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:02:19 (1713427339) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:02:34 (1713427354) targets are mounted 04:02:35 (1713427355) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d53e.replay-single-1/f has type file OK /mnt/lustre/d53e.replay-single-2/f has type file OK PASS 53e (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 04:02:44 (1713427364) fail_loc=0x119 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:02:49 (1713427369) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:03:03 (1713427383) targets are mounted 04:03:03 (1713427383) facet_failover done fail_loc=0 /mnt/lustre/d53f.replay-single-1/f has type file OK /mnt/lustre/d53f.replay-single-2/f has type file OK PASS 53f (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 04:03:12 (1713427392) fail_loc=0x119 fail_loc=0x80000115 fail_loc=0 Replay barrier on lustre-MDT0000 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:03:18 (1713427398) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:03:32 (1713427412) targets are mounted 04:03:32 (1713427412) facet_failover done /mnt/lustre/d53g.replay-single-1/f has type file OK /mnt/lustre/d53g.replay-single-2/f has type file OK PASS 53g (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 04:03:41 (1713427421) fail_loc=0x80000107 fail_loc=0x8000013b Replay barrier on lustre-MDT0000 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:03:47 (1713427427) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:04:02 (1713427442) targets are mounted 04:04:02 (1713427442) facet_failover done fail_loc=0 /mnt/lustre/d53h.replay-single-1/f has type file OK /mnt/lustre/d53h.replay-single-2/f has type file OK PASS 53h (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 04:04:12 (1713427452) fail_loc=0x8000012b fail_loc=0x0 rm: touch: cannot remove '/mnt/lustre/f55.replay-single'cannot touch '/mnt/lustre/f55.replay-single': No such file or directory : No such file or directory PASS 55 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 04:05:16 (1713427516) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1812 1285876 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:05:20 (1713427520) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:05:35 (1713427535) targets are mounted 04:05:35 (1713427535) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 56 (37s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 57: test recovery from llog for setattr op ========================================================== 04:05:55 (1713427555) fail_loc=0x8000012c UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1812 1285876 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:05:58 (1713427558) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:06:13 (1713427573) targets are mounted 04:06:13 (1713427573) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 oleg457-server: *.lustre-MDT0000.recovery_status status: COMPLETE Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. /mnt/lustre/f57.replay-single has type file OK fail_loc=0x0 PASS 57 (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 04:06:26 (1713427586) fail_loc=0x8000012c total: 2500 open/close in 4.88 seconds: 512.49 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1812 1285876 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:06:36 (1713427596) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:06:52 (1713427612) targets are mounted 04:06:52 (1713427612) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0x0 - unlinked 0 (time 1713427627 ; total 0 ; last 0) total: 2500 unlinks in 4 seconds: 625.000000 unlinks/second PASS 58a (48s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 58b: test replay of setxattr op ==== 04:07:16 (1713427636) Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2372 1285316 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1812 1285876 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:07:21 (1713427641) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:07:36 (1713427656) targets are mounted 04:07:36 (1713427656) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) oleg457-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid mgc.*.mgs_server_uuid in FULL state after 0 sec PASS 58b (30s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 04:07:48 (1713427668) Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre2 fail_val=0 fail_loc=0x123 fail_loc=0 fail_loc=0x119 fail_loc=0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) PASS 58c (129s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-single test_59 skipping ALWAYS excluded test 59 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 04:10:00 (1713427800) total: 200 open/close in 0.47 seconds: 428.01 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2228 1285460 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1812 1285876 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre - unlinked 0 (time 1713427805 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:10:06 (1713427806) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:10:21 (1713427821) targets are mounted 04:10:21 (1713427821) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1713427827 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second PASS 60 (30s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 04:10:32 (1713427832) total: 800 open/close in 2.67 seconds: 299.90 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2312 1285376 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1988 1285700 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210916 1% /mnt/lustre - unlinked 0 (time 1713427840 ; total 0 ; last 0) total: 800 unlinks in 1 seconds: 800.000000 unlinks/second fail_val=0 fail_loc=0x80000221 Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 04:10:43 (1713427843) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 04:10:59 (1713427859) targets are mounted 04:10:59 (1713427859) facet_failover done Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 04:11:10 (1713427870) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 04:11:23 (1713427883) targets are mounted 04:11:23 (1713427883) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 Can't lstat /mnt/lustre/d61a.replay-single/f61a.replay-single-*: No such file or directory PASS 61a (86s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 04:11:59 (1713427919) fail_loc=0x8000013a Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:12:01 (1713427921) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:12:14 (1713427934) targets are mounted 04:12:14 (1713427934) facet_failover done Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:12:25 (1713427945) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:12:38 (1713427958) targets are mounted 04:12:38 (1713427958) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0034952 s, 1.2 MB/s PASS 61b (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 04:12:47 (1713427967) fail_val=0 fail_loc=0x80000222 Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 04:12:59 (1713427979) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 04:13:12 (1713427992) targets are mounted 04:13:12 (1713427992) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_val=0 fail_loc=0x0 PASS 61c (30s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 04:13:19 (1713427999) Stopping /mnt/lustre-mds1 (opts:) on oleg457-server fail_loc=0x80000605 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Operation not supported pdsh@oleg457-client: oleg457-server: ssh exited with exit code 95 Start of /dev/mapper/mds1_flakey on mgs failed 95 fail_loc=0 Starting mgs: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 61d (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 62: don't mis-drop resent replay === 04:13:29 (1713428009) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2300 1285388 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1968 1285720 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3128 7210912 1% /mnt/lustre total: 25 open/close in 0.06 seconds: 403.65 ops/second fail_loc=0x80000707 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:13:34 (1713428014) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:13:47 (1713428027) targets are mounted 04:13:47 (1713428027) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 - unlinked 0 (time 1713428093 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second PASS 62 (85s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 65a: AT: verify early replies ====== 04:14:56 (1713428096) at_history=8 at_history=8 debug=other fail_val=11000 fail_loc=0x8000050a 00000100:00001000:0.0:1713428126.644375:0:14490:0:(client.c:537:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff88012b4c1180 x1796656273783104/t0(0) o101->lustre-MDT0000-mdc-ffff8800b42ec800@192.168.204.157@tcp:12/10 lens 664/66320 e 1 to 0 dl 1713428161 ref 2 fl Rpc:PQr/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 portal 12 : cur 36 worst 40 (at 1713426714, 1418s ago) 40 40 40 40 portal 29 : cur 5 worst 5 (at 1713425243, 2889s ago) 5 0 0 0 portal 23 : cur 5 worst 5 (at 1713425243, 2889s ago) 5 5 5 5 portal 30 : cur 5 worst 5 (at 1713425247, 2885s ago) 5 0 0 0 portal 17 : cur 5 worst 5 (at 1713425326, 2806s ago) 5 5 5 0 portal 13 : cur 5 worst 5 (at 1713425697, 2435s ago) 5 5 0 0 portal 12 : cur 5 worst 40 (at 1713426714, 1427s ago) 40 40 40 40 portal 29 : cur 5 worst 5 (at 1713425243, 2898s ago) 5 0 0 0 portal 23 : cur 5 worst 5 (at 1713425243, 2898s ago) 5 5 5 5 portal 30 : cur 5 worst 5 (at 1713425247, 2894s ago) 5 0 0 0 portal 17 : cur 5 worst 5 (at 1713425326, 2815s ago) 5 5 5 0 portal 13 : cur 5 worst 5 (at 1713425697, 2444s ago) 5 5 0 0 PASS 65a (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 04:15:44 (1713428144) at_history=8 at_history=8 debug=other trace fail_val=11 fail_loc=0x224 fail_loc=0 00000100:00001000:1.0:1713428174.979965:0:1953:0:(client.c:537:ptlrpc_at_recv_early_reply()) @@@ Early reply #1, new deadline in 35s (25s) req@ffff8800a85caa00 x1796656273793344/t0(0) o4->lustre-OST0000-osc-ffff8800b42ec800@192.168.204.157@tcp:6/4 lens 4584/448 e 1 to 0 dl 1713428209 ref 2 fl Rpc:Qr/200/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck portal 28 : cur 5 worst 5 (at 1713425243, 2937s ago) 5 5 5 5 portal 7 : cur 5 worst 5 (at 1713425244, 2936s ago) 5 5 5 5 portal 6 : cur 36 worst 36 (at 1713428179, 1s ago) 36 0 0 0 portal 17 : cur 5 worst 5 (at 1713425506, 2674s ago) 5 5 5 5 PASS 65b (38s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 04:16:24 (1713428184) at_history=8 at_history=8 portal 12 : cur 5 worst 40 (at 1713426714, 1492s ago) 40 40 40 40 fail_val=5000 fail_loc=0x8000050a portal 12 : cur 5 worst 40 (at 1713426714, 1498s ago) 40 40 40 40 fail_val=10000 fail_loc=0x8000050a portal 12 : cur 36 worst 40 (at 1713426714, 1509s ago) 40 40 40 40 fail_loc=0 portal 12 : cur 5 worst 40 (at 1713426714, 1519s ago) 40 40 40 40 Current MDT timeout 5, worst 40 PASS 66a (50s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 66b: AT: verify net latency adjusts ========================================================== 04:17:16 (1713428236) at_history=8 at_history=8 fail_val=10 fail_loc=0x50c fail_loc=0 network timeout orig 5, cur 10, worst 10 PASS 66b (85s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 04:18:42 (1713428322) at_history=8 at_history=8 fail_val=400 fail_loc=0x50a fail_loc=0 0 osc reconnect attempts on gradual slow PASS 67a (74s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 04:19:57 (1713428397) at_history=8 at_history=8 Creating to objid 5057 on ost lustre-OST0000... fail_val=20000 fail_loc=0x80000223 total: 17 open/close in 0.06 seconds: 262.90 ops/second Connected clients: oleg457-client.virtnet oleg457-client.virtnet service : cur 5 worst 5 (at 1713424985, 3437s ago) 1 0 1 1 phase 2 0 osc reconnect attempts on instant slow fail_loc=0x80000223 fail_loc=0 Connected clients: oleg457-client.virtnet oleg457-client.virtnet service : cur 5 worst 5 (at 1713424985, 3438s ago) 1 1 0 1 0 osc reconnect attempts on 2nd slow PASS 67b (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 68: AT: verify slowing locks ======= 04:20:26 (1713428426) at_history=8 at_history=8 /home/green/git/lustre-release/lustre/tests/replay-single.sh: line 1968: $ldlm_enqueue_min: ambiguous redirect fail_val=19 fail_loc=0x80000312 fail_val=25 fail_loc=0x80000312 fail_loc=0 /home/green/git/lustre-release/lustre/tests/replay-single.sh: line 1983: $ldlm_enqueue_min: ambiguous redirect PASS 68 (70s) debug_raw_pointers=0 debug_raw_pointers=0 Cleaning up AT ... debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70a: check multi client t-f ======== 04:21:38 (1713428498) SKIP: replay-single test_70a Need two or more clients, have 1 SKIP 70a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 04:21:41 (1713428501) Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg457-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=21605 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1285564 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3592224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26128 3556172 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27712 7148396 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:21:49 (1713428509) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:22:05 (1713428525) targets are mounted 04:22:05 (1713428525) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2092 1285596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3592968 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26392 3569804 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28204 7162772 1% /mnt/lustre test_70b fail mds2 2 times oleg457-client.virtnet: looking for dbench program oleg457-client.virtnet: /usr/bin/dbench oleg457-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg457-client.virtnet oleg457-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg457-client.virtnet' oleg457-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg457-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg457-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg457-client.virtnet at Thu Apr 18 04:21:42 EDT 2024 oleg457-client.virtnet: waiting for dbench pid 21629 oleg457-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg457-client.virtnet: oleg457-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg457-client.virtnet: failed to create barrier semaphore oleg457-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg457-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg457-client.virtnet: releasing clients oleg457-client.virtnet: 1 480 16.69 MB/sec warmup 1 sec latency 15.306 ms oleg457-client.virtnet: 1 1074 11.46 MB/sec warmup 2 sec latency 12.161 ms oleg457-client.virtnet: 1 1706 8.74 MB/sec warmup 3 sec latency 11.690 ms oleg457-client.virtnet: 1 2356 6.97 MB/sec warmup 4 sec latency 15.360 ms oleg457-client.virtnet: 1 2918 6.69 MB/sec warmup 5 sec latency 15.430 ms oleg457-client.virtnet: 1 3592 6.41 MB/sec warmup 6 sec latency 14.530 ms oleg457-client.virtnet: 1 3931 5.72 MB/sec warmup 7 sec latency 18.715 ms oleg457-client.virtnet: 1 4083 5.01 MB/sec warmup 8 sec latency 349.420 ms oleg457-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 1349.680 ms oleg457-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 2349.957 ms oleg457-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 3350.185 ms oleg457-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 4350.505 ms oleg457-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 5350.774 ms oleg457-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 6351.022 ms oleg457-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 7351.379 ms oleg457-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 8351.589 ms oleg457-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 9351.785 ms oleg457-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 10351.930 ms oleg457-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 11352.113 ms oleg457-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 12352.374 ms oleg457-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 13352.559 ms oleg457-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 14352.753 ms oleg457-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 15353.053 ms oleg457-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 16353.416 ms oleg457-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 17353.761 ms oleg457-client.virtnet: 1 4314 1.56 MB/sec warmup 26 sec latency 17789.666 ms oleg457-client.virtnet: 1 4667 1.56 MB/sec warmup 27 sec latency 19.118 ms oleg457-client.virtnet: 1 5143 1.62 MB/sec warmup 28 sec latency 15.402 ms oleg457-client.virtnet: 1 5759 1.58 MB/sec warmup 29 sec latency 11.002 ms oleg457-client.virtnet: 1 6319 1.70 MB/sec warmup 30 sec latency 15.802 ms oleg457-client.virtnet: 1 7134 1.86 MB/sec warmup 31 sec latency 14.682 ms oleg457-client.virtnet: 1 7503 1.85 MB/sec warmup 32 sec latency 15.870 ms oleg457-client.virtnet: 1 7824 1.80 MB/sec warmup 33 sec latency 19.0Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:22:17 (1713428537) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:22:31 (1713428551) targets are mounted 04:22:31 (1713428551) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13136 3592964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36380 3570020 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49516 7162984 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:22:44 (1713428564) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:23:00 (1713428580) targets are mounted 04:23:00 (1713428580) facet_failover done 99 ms oleg457-client.virtnet: 1 8258 1.80 MB/sec warmup 34 sec latency 169.316 ms oleg457-client.virtnet: 1 8258 1.75 MB/sec warmup 35 sec latency 1169.501 ms oleg457-client.virtnet: 1 8258 1.70 MB/sec warmup 36 sec latency 2169.738 ms oleg457-client.virtnet: 1 8258 1.65 MB/sec warmup 37 sec latency 3170.067 ms oleg457-client.virtnet: 1 8258 1.61 MB/sec warmup 38 sec latency 4170.349 ms oleg457-client.virtnet: 1 8258 1.57 MB/sec warmup 39 sec latency 5170.671 ms oleg457-client.virtnet: 1 8258 1.53 MB/sec warmup 40 sec latency 6170.936 ms oleg457-client.virtnet: 1 8258 1.49 MB/sec warmup 41 sec latency 7171.182 ms oleg457-client.virtnet: 1 8258 1.46 MB/sec warmup 42 sec latency 8171.445 ms oleg457-client.virtnet: 1 8258 1.42 MB/sec warmup 43 sec latency 9171.672 ms oleg457-client.virtnet: 1 8258 1.39 MB/sec warmup 44 sec latency 10171.896 ms oleg457-client.virtnet: 1 8258 1.36 MB/sec warmup 45 sec latency 11172.155 ms oleg457-client.virtnet: 1 8258 1.33 MB/sec warmup 46 sec latency 12172.411 ms oleg457-client.virtnet: 1 8258 1.30 MB/sec warmup 47 sec latency 13172.615 ms oleg457-client.virtnet: 1 8258 1.28 MB/sec warmup 48 sec latency 14172.784 ms oleg457-client.virtnet: 1 8258 1.25 MB/sec warmup 49 sec latency 15172.982 ms oleg457-client.virtnet: 1 8258 1.22 MB/sec warmup 50 sec latency 16173.236 ms oleg457-client.virtnet: 1 8258 1.20 MB/sec warmup 51 sec latency 17173.657 ms oleg457-client.virtnet: 1 8258 1.18 MB/sec warmup 52 sec latency 18173.913 ms oleg457-client.virtnet: 1 8258 1.16 MB/sec warmup 53 sec latency 19174.144 ms oleg457-client.virtnet: 1 8647 1.19 MB/sec warmup 54 sec latency 19426.326 ms oleg457-client.virtnet: 1 9107 1.18 MB/sec warmup 55 sec latency 17.065 ms oleg457-client.virtnet: 1 9657 1.20 MB/sec warmup 56 sec latency 12.320 ms oleg457-client.virtnet: 1 10440 1.29 MB/sec warmup 57 sec latency 15.578 ms oleg457-client.virtnet: 1 10974 1.34 MB/sec warmup 58 sec latency 11.706 ms oleg457-client.virtnet: 1 11357 1.33 MB/sec warmup 59 sec latency 19.289 ms oleg457-client.virtnet: 1 12287 4.26 MB/sec execute 1 sec latency 16.173 ms oleg457-client.virtnet: 1 12884 2.44 MB/sec execute 2 sec latency 9.538 ms oleg457-client.virtnet: 1 13261 2.37 MB/sec execute 3 sec latency 15.962 ms oleg457-client.virtnet: 1 14268 4.17 MB/sec execute 4 sec latency 10.762 ms oleg457-client.virtnet: 1 14554 3.63 MB/sec execute 5 sec latency 16.000 ms oleg457-client.virtnet: 1 15027 3.12 MB/sec execute 6 sec latency 13.952 ms oleg457-client.virtnet: 1 15404 2.89 MB/sec execute 7 sec latency 16.573 ms oleg457-client.virtnet: 1 15937 2.94 MB/sec execute 8 sec latency 15.692 ms oleg457-client.virtnet: 1 16563 2.79 MB/sec execute 9 sec latency 11.247 ms oleg457-client.virtnet: 1 17084 2.99 MB/sec execute 10 sec latency 15.154 ms oleg457-client.virtnet: 1 17715 3.23 MB/sec execute 11 sec latency 16.522 ms oleg457-client.virtnet: 1 18020 3.08 MB/sec execute 12 sec latency 15.894 ms oleg457-client.virtnet: 1 18259 2.87 MB/sec execute 13 sec latency 15.673 ms oleg457-client.virtnet: 1 18621 2.71 MB/sec execute 14 sec latency 23.024 ms oleg457-client.virtnet: 1 18964 2.63 MB/sec execute 15 sec latency 15.496 ms oleg457-client.virtnet: 1 19374 2.66 MB/sec execute 16 sec latency 15.696 ms oleg457-client.virtnet: 1 19751 2.52 MB/sec execute 17 sec latency 16.470 ms oleg457-client.virtnet: 1 20152 2.46 MB/sec execute 18 sec latency 13.988 ms oleg457-client.virtnet: 1 20657 2.56 MB/sec execuoleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2048 1285640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13068 3592960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36556 3569108 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7162068 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:23:12 (1713428592) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:23:28 (1713428608) targets are mounted 04:23:28 (1713428608) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13068 3591884 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36532 3567656 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49600 7159540 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:23:41 (1713428621) shut down te 19 sec latency 15.811 ms oleg457-client.virtnet: 1 21284 2.74 MB/sec execute 20 sec latency 15.615 ms oleg457-client.virtnet: 1 21601 2.67 MB/sec execute 21 sec latency 16.303 ms oleg457-client.virtnet: 1 21861 2.57 MB/sec execute 22 sec latency 14.167 ms oleg457-client.virtnet: 1 22208 2.49 MB/sec execute 23 sec latency 21.481 ms oleg457-client.virtnet: 1 22562 2.44 MB/sec execute 24 sec latency 15.735 ms oleg457-client.virtnet: 1 23053 2.47 MB/sec execute 25 sec latency 16.744 ms oleg457-client.virtnet: 1 23425 2.39 MB/sec execute 26 sec latency 12.996 ms oleg457-client.virtnet: 1 23783 2.36 MB/sec execute 27 sec latency 16.345 ms oleg457-client.virtnet: 1 24335 2.47 MB/sec execute 28 sec latency 16.255 ms oleg457-client.virtnet: 1 24960 2.56 MB/sec execute 29 sec latency 17.374 ms oleg457-client.virtnet: 1 25039 2.48 MB/sec execute 30 sec latency 721.537 ms oleg457-client.virtnet: 1 25039 2.40 MB/sec execute 31 sec latency 1721.817 ms oleg457-client.virtnet: 1 25039 2.32 MB/sec execute 32 sec latency 2722.058 ms oleg457-client.virtnet: 1 25039 2.25 MB/sec execute 33 sec latency 3722.282 ms oleg457-client.virtnet: 1 25039 2.19 MB/sec execute 34 sec latency 4722.552 ms oleg457-client.virtnet: 1 25039 2.12 MB/sec execute 35 sec latency 5722.749 ms oleg457-client.virtnet: 1 25039 2.06 MB/sec execute 36 sec latency 6723.050 ms oleg457-client.virtnet: 1 25039 2.01 MB/sec execute 37 sec latency 7723.264 ms oleg457-client.virtnet: 1 25039 1.96 MB/sec execute 38 sec latency 8723.540 ms oleg457-client.virtnet: 1 25039 1.90 MB/sec execute 39 sec latency 9723.775 ms oleg457-client.virtnet: 1 25039 1.86 MB/sec execute 40 sec latency 10724.027 ms oleg457-client.virtnet: 1 25039 1.81 MB/sec execute 41 sec latency 11724.291 ms oleg457-client.virtnet: 1 25039 1.77 MB/sec execute 42 sec latency 12724.497 ms oleg457-client.virtnet: 1 25039 1.73 MB/sec execute 43 sec latency 13724.689 ms oleg457-client.virtnet: 1 25039 1.69 MB/sec execute 44 sec latency 14724.900 ms oleg457-client.virtnet: 1 25039 1.65 MB/sec execute 45 sec latency 15725.132 ms oleg457-client.virtnet: 1 25039 1.62 MB/sec execute 46 sec latency 16725.370 ms oleg457-client.virtnet: 1 25039 1.58 MB/sec execute 47 sec latency 17725.599 ms oleg457-client.virtnet: 1 25039 1.55 MB/sec execute 48 sec latency 18725.852 ms oleg457-client.virtnet: 1 25039 1.52 MB/sec execute 49 sec latency 19726.057 ms oleg457-client.virtnet: 1 25274 1.51 MB/sec execute 50 sec latency 19825.998 ms oleg457-client.virtnet: 1 25585 1.49 MB/sec execute 51 sec latency 17.081 ms oleg457-client.virtnet: 1 26058 1.49 MB/sec execute 52 sec latency 14.920 ms oleg457-client.virtnet: 1 26449 1.52 MB/sec execute 53 sec latency 18.327 ms oleg457-client.virtnet: 1 26749 1.50 MB/sec execute 54 sec latency 16.871 ms oleg457-client.virtnet: 1 27177 1.50 MB/sec execute 55 sec latency 11.635 ms oleg457-client.virtnet: 1 27742 1.55 MB/sec execute 56 sec latency 16.508 ms oleg457-client.virtnet: 1 28385 1.63 MB/sec execute 57 sec latency 15.123 ms oleg457-client.virtnet: 1 28718 1.63 MB/sec execute 58 sec latency 15.523 ms oleg457-client.virtnet: 1 29005 1.61 MB/sec execute 59 sec latency 15.404 ms oleg457-client.virtnet: 1 29362 1.59 MB/sec execute 60 sec latency 16.124 ms oleg457-client.virtnet: 1 29827 1.64 MB/sec execute 61 sec latency 15.974 ms oleg457-client.virtnet: 1 30146 1.61 MB/sec execute 62 sec latency 17.982 ms oleg457-client.virtnet: 1 30518 1.59 MB/sec execute 63 sec latency 13.755 ms oleg457-clFailover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:23:57 (1713428637) targets are mounted 04:23:57 (1713428637) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2044 1285644 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13228 3592000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36424 3569896 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49652 7161896 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:24:10 (1713428650) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:24:25 (1713428665) targets are mounted 04:24:25 (1713428665) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec ient.virtnet: 1 30878 1.59 MB/sec execute 64 sec latency 15.067 ms oleg457-client.virtnet: 1 31440 1.65 MB/sec execute 65 sec latency 15.982 ms oleg457-client.virtnet: 1 32297 1.72 MB/sec execute 66 sec latency 15.317 ms oleg457-client.virtnet: 1 32584 1.70 MB/sec execute 67 sec latency 16.342 ms oleg457-client.virtnet: 1 32888 1.69 MB/sec execute 68 sec latency 23.730 ms oleg457-client.virtnet: 1 33234 1.68 MB/sec execute 69 sec latency 16.116 ms oleg457-client.virtnet: 1 33626 1.70 MB/sec execute 70 sec latency 15.960 ms oleg457-client.virtnet: 1 33976 1.68 MB/sec execute 71 sec latency 18.814 ms oleg457-client.virtnet: 1 34384 1.68 MB/sec execute 72 sec latency 15.173 ms oleg457-client.virtnet: 1 34955 1.73 MB/sec execute 73 sec latency 15.245 ms oleg457-client.virtnet: 1 35583 1.78 MB/sec execute 74 sec latency 14.705 ms oleg457-client.virtnet: 1 35864 1.77 MB/sec execute 75 sec latency 16.632 ms oleg457-client.virtnet: 1 36176 1.75 MB/sec execute 76 sec latency 15.955 ms oleg457-client.virtnet: 1 36454 1.74 MB/sec execute 77 sec latency 20.286 ms oleg457-client.virtnet: 1 36811 1.74 MB/sec execute 78 sec latency 16.234 ms oleg457-client.virtnet: 1 37200 1.75 MB/sec execute 79 sec latency 16.256 ms oleg457-client.virtnet: 1 37534 1.73 MB/sec execute 80 sec latency 16.626 ms oleg457-client.virtnet: 1 37926 1.73 MB/sec execute 81 sec latency 14.660 ms oleg457-client.virtnet: 1 38448 1.77 MB/sec execute 82 sec latency 16.171 ms oleg457-client.virtnet: 1 39173 1.82 MB/sec execute 83 sec latency 15.025 ms oleg457-client.virtnet: 1 39447 1.81 MB/sec execute 84 sec latency 15.706 ms oleg457-client.virtnet: 1 39764 1.79 MB/sec execute 85 sec latency 19.785 ms oleg457-client.virtnet: 1 40048 1.78 MB/sec execute 86 sec latency 16.430 ms oleg457-client.virtnet: 1 40229 1.77 MB/sec execute 87 sec latency 507.482 ms oleg457-client.virtnet: 1 40229 1.75 MB/sec execute 88 sec latency 1507.716 ms oleg457-client.virtnet: 1 40229 1.73 MB/sec execute 89 sec latency 2507.957 ms oleg457-client.virtnet: 1 40229 1.71 MB/sec execute 90 sec latency 3508.152 ms oleg457-client.virtnet: 1 40229 1.69 MB/sec execute 91 sec latency 4508.443 ms oleg457-client.virtnet: 1 40229 1.67 MB/sec execute 92 sec latency 5508.697 ms oleg457-client.virtnet: 1 40229 1.66 MB/sec execute 93 sec latency 6508.966 ms oleg457-client.virtnet: 1 40229 1.64 MB/sec execute 94 sec latency 7509.171 ms oleg457-client.virtnet: 1 40229 1.62 MB/sec execute 95 sec latency 8509.446 ms oleg457-client.virtnet: 1 40229 1.61 MB/sec execute 96 sec latency 9509.727 ms oleg457-client.virtnet: 1 40229 1.59 MB/sec execute 97 sec latency 10509.920 ms oleg457-client.virtnet: 1 40229 1.57 MB/sec execute 98 sec latency 11510.066 ms oleg457-client.virtnet: 1 40229 1.56 MB/sec execute 99 sec latency 12510.256 ms oleg457-client.virtnet: 1 40229 1.54 MB/sec execute 100 sec latency 13510.470 ms oleg457-client.virtnet: 1 40229 1.53 MB/sec execute 101 sec latency 14511.968 ms oleg457-client.virtnet: 1 40229 1.51 MB/sec execute 102 sec latency 15512.176 ms oleg457-client.virtnet: 1 40229 1.50 MB/sec execute 103 sec latency 16512.435 ms oleg457-client.virtnet: 1 40229 1.48 MB/sec execute 104 sec latency 17512.621 ms oleg457-client.virtnet: 1 40229 1.47 MB/sec execute 105 sec latency 18512.835 ms oleg457-client.virtnet: 1 40229 1.45 MB/sec execute 106 sec latency 19513.082 ms oleg457-client.virtnet: 1 40484 1.47 MB/sec execute 107 sec latency 19946.969 ms oleg457-client.virtnet: 1 40775 1.46 MBUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13244 3592792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36512 3569976 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49756 7162768 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:24:39 (1713428679) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:24:54 (1713428694) targets are mounted 04:24:54 (1713428694) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13116 3590816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36476 3569748 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49592 7160564 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:25:07 (1713428707) shut down /sec execute 108 sec latency 17.624 ms oleg457-client.virtnet: 1 41120 1.45 MB/sec execute 109 sec latency 12.155 ms oleg457-client.virtnet: 1 41485 1.45 MB/sec execute 110 sec latency 16.171 ms oleg457-client.virtnet: 1 42047 1.48 MB/sec execute 111 sec latency 16.397 ms oleg457-client.virtnet: 1 42662 1.52 MB/sec execute 112 sec latency 16.620 ms oleg457-client.virtnet: 1 42928 1.51 MB/sec execute 113 sec latency 17.603 ms oleg457-client.virtnet: 1 43203 1.50 MB/sec execute 114 sec latency 16.369 ms oleg457-client.virtnet: 1 43520 1.50 MB/sec execute 115 sec latency 18.385 ms oleg457-client.virtnet: 1 43855 1.49 MB/sec execute 116 sec latency 18.338 ms oleg457-client.virtnet: 1 44246 1.51 MB/sec execute 117 sec latency 15.712 ms oleg457-client.virtnet: 1 44603 1.50 MB/sec execute 118 sec latency 16.948 ms oleg457-client.virtnet: 1 44993 1.50 MB/sec execute 119 sec latency 16.404 ms oleg457-client.virtnet: 1 45513 1.53 MB/sec execute 120 sec latency 15.586 ms oleg457-client.virtnet: 1 46142 1.56 MB/sec execute 121 sec latency 16.191 ms oleg457-client.virtnet: 1 46527 1.56 MB/sec execute 122 sec latency 15.233 ms oleg457-client.virtnet: 1 46831 1.55 MB/sec execute 123 sec latency 22.856 ms oleg457-client.virtnet: 1 47126 1.54 MB/sec execute 124 sec latency 16.146 ms oleg457-client.virtnet: 1 47543 1.56 MB/sec execute 125 sec latency 18.098 ms oleg457-client.virtnet: 1 47843 1.55 MB/sec execute 126 sec latency 18.512 ms oleg457-client.virtnet: 1 48213 1.54 MB/sec execute 127 sec latency 10.823 ms oleg457-client.virtnet: 1 48601 1.54 MB/sec execute 128 sec latency 15.624 ms oleg457-client.virtnet: 1 49551 1.59 MB/sec execute 129 sec latency 13.912 ms oleg457-client.virtnet: 1 49977 1.61 MB/sec execute 130 sec latency 15.802 ms oleg457-client.virtnet: 1 50219 1.60 MB/sec execute 131 sec latency 15.899 ms oleg457-client.virtnet: 1 50571 1.59 MB/sec execute 132 sec latency 15.797 ms oleg457-client.virtnet: 1 50894 1.59 MB/sec execute 133 sec latency 15.976 ms oleg457-client.virtnet: 1 51375 1.60 MB/sec execute 134 sec latency 15.725 ms oleg457-client.virtnet: 1 51727 1.59 MB/sec execute 135 sec latency 14.216 ms oleg457-client.virtnet: 1 52126 1.59 MB/sec execute 136 sec latency 15.371 ms oleg457-client.virtnet: 1 52669 1.62 MB/sec execute 137 sec latency 15.961 ms oleg457-client.virtnet: 1 53270 1.64 MB/sec execute 138 sec latency 16.076 ms oleg457-client.virtnet: 1 53571 1.64 MB/sec execute 139 sec latency 15.316 ms oleg457-client.virtnet: 1 53839 1.63 MB/sec execute 140 sec latency 17.006 ms oleg457-client.virtnet: 1 54037 1.62 MB/sec execute 141 sec latency 369.500 ms oleg457-client.virtnet: 1 54286 1.62 MB/sec execute 142 sec latency 584.665 ms oleg457-client.virtnet: 1 54690 1.63 MB/sec execute 143 sec latency 15.543 ms oleg457-client.virtnet: 1 54967 1.62 MB/sec execute 144 sec latency 84.269 ms oleg457-client.virtnet: 1 54967 1.61 MB/sec execute 145 sec latency 1084.550 ms oleg457-client.virtnet: 1 54967 1.60 MB/sec execute 146 sec latency 2084.754 ms oleg457-client.virtnet: 1 54967 1.59 MB/sec execute 147 sec latency 3084.981 ms oleg457-client.virtnet: 1 54967 1.58 MB/sec execute 148 sec latency 4085.213 ms oleg457-client.virtnet: 1 54967 1.57 MB/sec execute 149 sec latency 5085.363 ms oleg457-client.virtnet: 1 54967 1.56 MB/sec execute 150 sec latency 6085.557 ms oleg457-client.virtnet: 1 54967 1.55 MB/sec execute 151 sec latency 7085.764 ms oleg457-client.virtnet: 1 54967 1.54 MB/sec execute 152 sec latency 8086.030 ms oleg457-client.virtnet: 1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:25:23 (1713428723) targets are mounted 04:25:23 (1713428723) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13112 3592552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36384 3567960 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7160512 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:25:36 (1713428736) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:25:51 (1713428751) targets are mounted 04:25:51 (1713428751) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593084 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36544 3569468 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7162552 1% /mnt/lustre 54967 1.53 MB/sec execute 153 sec latency 9086.291 ms oleg457-client.virtnet: 1 54967 1.52 MB/sec execute 154 sec latency 10086.523 ms oleg457-client.virtnet: 1 54967 1.51 MB/sec execute 155 sec latency 11086.717 ms oleg457-client.virtnet: 1 54967 1.50 MB/sec execute 156 sec latency 12086.883 ms oleg457-client.virtnet: 1 54967 1.49 MB/sec execute 157 sec latency 13087.143 ms oleg457-client.virtnet: 1 54967 1.48 MB/sec execute 158 sec latency 14087.382 ms oleg457-client.virtnet: 1 54967 1.47 MB/sec execute 159 sec latency 15087.640 ms oleg457-client.virtnet: 1 54967 1.46 MB/sec execute 160 sec latency 16087.850 ms oleg457-client.virtnet: 1 54967 1.45 MB/sec execute 161 sec latency 17088.125 ms oleg457-client.virtnet: 1 54967 1.44 MB/sec execute 162 sec latency 18088.403 ms oleg457-client.virtnet: 1 54967 1.43 MB/sec execute 163 sec latency 19088.822 ms oleg457-client.virtnet: 1 55044 1.43 MB/sec execute 164 sec latency 19837.150 ms oleg457-client.virtnet: 1 55436 1.42 MB/sec execute 165 sec latency 12.650 ms oleg457-client.virtnet: 1 55842 1.43 MB/sec execute 166 sec latency 16.819 ms oleg457-client.virtnet: 1 56587 1.45 MB/sec execute 167 sec latency 16.097 ms oleg457-client.virtnet: 1 56997 1.47 MB/sec execute 168 sec latency 15.214 ms oleg457-client.virtnet: 1 57243 1.47 MB/sec execute 169 sec latency 18.184 ms oleg457-client.virtnet: 1 57569 1.46 MB/sec execute 170 sec latency 19.133 ms oleg457-client.virtnet: 1 57888 1.46 MB/sec execute 171 sec latency 17.102 ms oleg457-client.virtnet: 1 58284 1.47 MB/sec execute 172 sec latency 15.974 ms oleg457-client.virtnet: 1 58607 1.46 MB/sec execute 173 sec latency 16.890 ms oleg457-client.virtnet: 1 58992 1.46 MB/sec execute 174 sec latency 11.252 ms oleg457-client.virtnet: 1 59376 1.46 MB/sec execute 175 sec latency 16.125 ms oleg457-client.virtnet: 1 60119 1.49 MB/sec execute 176 sec latency 15.499 ms oleg457-client.virtnet: 1 60547 1.51 MB/sec execute 177 sec latency 15.436 ms oleg457-client.virtnet: 1 60778 1.50 MB/sec execute 178 sec latency 15.683 ms oleg457-client.virtnet: 1 61102 1.49 MB/sec execute 179 sec latency 19.629 ms oleg457-client.virtnet: 1 61414 1.49 MB/sec execute 180 sec latency 16.017 ms oleg457-client.virtnet: 1 61822 1.50 MB/sec execute 181 sec latency 16.096 ms oleg457-client.virtnet: 1 62332 1.50 MB/sec execute 182 sec latency 13.091 ms oleg457-client.virtnet: 1 62748 1.50 MB/sec execute 183 sec latency 15.295 ms oleg457-client.virtnet: 1 63263 1.52 MB/sec execute 184 sec latency 16.405 ms oleg457-client.virtnet: 1 63890 1.54 MB/sec execute 185 sec latency 15.885 ms oleg457-client.virtnet: 1 64179 1.54 MB/sec execute 186 sec latency 16.463 ms oleg457-client.virtnet: 1 64548 1.53 MB/sec execute 187 sec latency 15.684 ms oleg457-client.virtnet: 1 65076 1.53 MB/sec execute 188 sec latency 13.108 ms oleg457-client.virtnet: 1 65473 1.54 MB/sec execute 189 sec latency 15.292 ms oleg457-client.virtnet: 1 65800 1.53 MB/sec execute 190 sec latency 17.476 ms oleg457-client.virtnet: 1 66222 1.53 MB/sec execute 191 sec latency 13.078 ms oleg457-client.virtnet: 1 66648 1.55 MB/sec execute 192 sec latency 16.137 ms oleg457-client.virtnet: 1 67363 1.57 MB/sec execute 193 sec latency 15.468 ms oleg457-client.virtnet: 1 67697 1.57 MB/sec execute 194 sec latency 15.916 ms oleg457-client.virtnet: 1 68150 1.57 MB/sec execute 195 sec latency 14.078 ms oleg457-client.virtnet: 1 68651 1.57 MB/sec execute 196 sec latency 14.053 ms oleg457-client.virtnet: 1 69406 1.58 MB/sec execute 197 sec latency 9.072 ms test_70b fail mds2 10 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:26:03 (1713428763) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:26:18 (1713428778) targets are mounted 04:26:18 (1713428778) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593108 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36608 3569876 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49552 7162984 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:26:31 (1713428791) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-client.virtnet: 1 69821 1.58 MB/sec execute 198 sec latency 446.218 ms oleg457-client.virtnet: 1 70757 1.61 MB/sec execute 199 sec latency 12.526 ms oleg457-client.virtnet: 1 71273 1.62 MB/sec execute 200 sec latency 12.792 ms oleg457-client.virtnet: 1 71349 1.61 MB/sec execute 201 sec latency 811.868 ms oleg457-client.virtnet: 1 71349 1.60 MB/sec execute 202 sec latency 1812.031 ms oleg457-client.virtnet: 1 71349 1.60 MB/sec execute 203 sec latency 2812.266 ms oleg457-client.virtnet: 1 71349 1.59 MB/sec execute 204 sec latency 3812.502 ms oleg457-client.virtnet: 1 71349 1.58 MB/sec execute 205 sec latency 4812.700 ms oleg457-client.virtnet: 1 71349 1.57 MB/sec execute 206 sec latency 5812.918 ms oleg457-client.virtnet: 1 71349 1.57 MB/sec execute 207 sec latency 6813.153 ms oleg457-client.virtnet: 1 71349 1.56 MB/sec execute 208 sec latency 7813.404 ms oleg457-client.virtnet: 1 71349 1.55 MB/sec execute 209 sec latency 8813.716 ms oleg457-client.virtnet: 1 71349 1.54 MB/sec execute 210 sec latency 9813.937 ms oleg457-client.virtnet: 1 71349 1.54 MB/sec execute 211 sec latency 10814.070 ms oleg457-client.virtnet: 1 71349 1.53 MB/sec execute 212 sec latency 11814.212 ms oleg457-client.virtnet: 1 71349 1.52 MB/sec execute 213 sec latency 12814.402 ms oleg457-client.virtnet: 1 71349 1.51 MB/sec execute 214 sec latency 13814.596 ms oleg457-client.virtnet: 1 71349 1.51 MB/sec execute 215 sec latency 14814.771 ms oleg457-client.virtnet: 1 71349 1.50 MB/sec execute 216 sec latency 15815.006 ms oleg457-client.virtnet: 1 71349 1.49 MB/sec execute 217 sec latency 16815.182 ms oleg457-client.virtnet: 1 71349 1.49 MB/sec execute 218 sec latency 17815.457 ms oleg457-client.virtnet: 1 71349 1.48 MB/sec execute 219 sec latency 18815.699 ms oleg457-client.virtnet: 1 71464 1.47 MB/sec execute 220 sec latency 19298.377 ms oleg457-client.virtnet: 1 71760 1.47 MB/sec execute 221 sec latency 15.909 ms oleg457-client.virtnet: 1 72056 1.47 MB/sec execute 222 sec latency 22.066 ms oleg457-client.virtnet: 1 72444 1.48 MB/sec execute 223 sec latency 15.457 ms oleg457-client.virtnet: 1 72815 1.47 MB/sec execute 224 sec latency 17.694 ms oleg457-client.virtnet: 1 73432 1.47 MB/sec execute 225 sec latency 11.944 ms oleg457-client.virtnet: 1 74202 1.49 MB/sec execute 226 sec latency 14.924 ms oleg457-client.virtnet: 1 74739 1.51 MB/sec execute 227 sec latency 12.132 ms oleg457-client.virtnet: 1 74977 1.51 MB/sec execute 228 sec latency 15.837 ms oleg457-client.virtnet: 1 75322 1.50 MB/sec execute 229 sec latency 23.281 ms oleg457-client.virtnet: 1 75687 1.50 MB/sec execute 230 sec latency 16.497 ms oleg457-client.virtnet: 1 76398 1.51 MB/sec execute 231 sec latency 16.715 ms oleg457-client.virtnet: 1 76940 1.51 MB/sec execute 232 sec latency 18.560 ms oleg457-client.virtnet: 1 77810 1.53 MB/sec execute 233 sec latency 12.309 ms oleg457-client.virtnet: 1 78536 1.55 MB/sec execute 234 sec latency 7.986 ms oleg457-client.virtnet: 1 79194 1.55 MB/sec execute 235 sec latency 11.399 ms oleg457-client.virtnet: 1 79916 1.56 MB/sec execute 236 sec latency 10.879 ms oleg457-client.virtnet: 1 80683 1.56 MB/sec execute 237 sec latency 9.337 ms oleg457-client.virtnet: 1 81779 1.60 MB/sec execute 238 sec latency 8.773 ms oleg457-client.virtnet: 1 82107 1.60 MB/sec execute 239 sec latency 15.617 ms oleg457-client.virtnet: 1 82635 1.59 MB/sec execute 240 sec latency 12.814 ms oleg457-client.virtnet: 1 83368 1.61 MB/sec execute 241 sec latency 9.327 ms oleg457-client.virtnet: 1 84104 1.6oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:26:45 (1713428805) targets are mounted 04:26:45 (1713428805) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 MB/sec execute 242 sec latency 7.937 ms oleg457-client.virtnet: 1 85286 1.64 MB/sec execute 243 sec latency 8.324 ms oleg457-client.virtnet: 1 85880 1.64 MB/sec execute 244 sec latency 10.202 ms oleg457-client.virtnet: 1 86608 1.66 MB/sec execute 245 sec latency 14.202 ms oleg457-client.virtnet: 1 87316 1.65 MB/sec execute 246 sec latency 8.339 ms oleg457-client.virtnet: 1 87785 1.65 MB/sec execute 247 sec latency 16.391 ms oleg457-client.virtnet: 1 88934 1.69 MB/sec execute 248 sec latency 7.993 ms oleg457-client.virtnet: 1 89318 1.69 MB/sec execute 249 sec latency 14.518 ms oleg457-client.virtnet: 1 89660 1.68 MB/sec execute 250 sec latency 19.100 ms oleg457-client.virtnet: 1 90118 1.69 MB/sec execute 251 sec latency 18.710 ms oleg457-client.virtnet: 1 90536 1.69 MB/sec execute 252 sec latency 14.709 ms oleg457-client.virtnet: 1 90979 1.68 MB/sec execute 253 sec latency 11.095 ms oleg457-client.virtnet: 1 91626 1.70 MB/sec execute 254 sec latency 16.564 ms oleg457-client.virtnet: 1 92259 1.71 MB/sec execute 255 sec latency 16.961 ms oleg457-client.virtnet: 1 92563 1.71 MB/sec execute 256 sec latency 16.683 ms oleg457-client.virtnet: 1 92956 1.71 MB/sec execute 257 sec latency 16.567 ms oleg457-client.virtnet: 1 93682 1.72 MB/sec execute 258 sec latency 8.588 ms oleg457-client.virtnet: 1 94236 1.72 MB/sec execute 259 sec latency 16.802 ms oleg457-client.virtnet: 1 94824 1.72 MB/sec execute 260 sec latency 11.263 ms oleg457-client.virtnet: 1 95802 1.75 MB/sec execute 261 sec latency 11.573 ms oleg457-client.virtnet: 1 96119 1.75 MB/sec execute 262 sec latency 15.730 ms oleg457-client.virtnet: 1 96568 1.74 MB/sec execute 263 sec latency 16.260 ms oleg457-client.virtnet: 1 97325 1.75 MB/sec execute 264 sec latency 8.165 ms oleg457-client.virtnet: 1 97876 1.75 MB/sec execute 265 sec latency 10.509 ms oleg457-client.virtnet: 1 98259 1.75 MB/sec execute 266 sec latency 15.540 ms oleg457-client.virtnet: 1 98827 1.76 MB/sec execute 267 sec latency 15.565 ms oleg457-client.virtnet: 1 99578 1.78 MB/sec execute 268 sec latency 14.556 ms oleg457-client.virtnet: 1 100065 1.77 MB/sec execute 269 sec latency 11.368 ms oleg457-client.virtnet: 1 100796 1.79 MB/sec execute 270 sec latency 8.832 ms oleg457-client.virtnet: 1 101548 1.78 MB/sec execute 271 sec latency 8.509 ms oleg457-client.virtnet: 1 102698 1.81 MB/sec execute 272 sec latency 7.624 ms oleg457-client.virtnet: 1 103347 1.82 MB/sec execute 273 sec latency 12.653 ms oleg457-client.virtnet: 1 103978 1.82 MB/sec execute 274 sec latency 11.089 ms oleg457-client.virtnet: 1 104677 1.82 MB/sec execute 275 sec latency 11.015 ms oleg457-client.virtnet: 1 105418 1.83 MB/sec execute 276 sec latency 8.348 ms oleg457-client.virtnet: 1 106585 1.86 MB/sec execute 277 sec latency 8.258 ms oleg457-client.virtnet: 1 107161 1.85 MB/sec execute 278 sec latency 11.422 ms oleg457-client.virtnet: 1 107858 1.87 MB/sec execute 279 sec latency 9.968 ms oleg457-client.virtnet: 1 108301 1.86 MB/sec execute 280 sec latency 13.428 ms oleg457-client.virtnet: 1 109036 1.86 MB/sec execute 281 sec latency 11.278 ms oleg457-client.virtnet: 1 110165 1.89 MB/sec execute 282 sec latency 12.862 ms oleg457-client.virtnet: 1 110663 1.89 MB/sec execute 283 sec latency 15.181 ms oleg457-client.virtnet: 1 111325 1.89 MB/sec execute 284 sec latency 11.052 ms oleg457-client.virtnet: 1 112083 1.90 MB/sec execute 285 sec latency 11.952 ms oleg457-client.virtnet: 1 113015 1.91 MB/sec execute 286 sec latency 8.926 ms oleg457-client.virtnet: 1 113937 1.93 MB/sec execute 287 sec latency 7.975 ms oleg457-client.virtnet: 1 114602 1.93 MB/sec execute 288 sec latency 11.246 ms oleg457-client.virtnet: 1 115371 1.93 MB/sec execute 289 sec latency 10.438 ms oleg457-client.virtnet: 1 116114 1.94 MB/sec execute 290 sec latency 9.886 ms oleg457-client.virtnet: 1 117209 1.96 MB/sec execute 291 sec latency 9.365 ms oleg457-client.virtnet: 1 117800 1.96 MB/sec execute 292 sec latency 9.319 ms oleg457-client.virtnet: 1 118535 1.97 MB/sec execute 293 sec latency 8.585 ms oleg457-client.virtnet: 1 119274 1.97 MB/sec execute 294 sec latency 8.670 ms oleg457-client.virtnet: 1 120404 1.99 MB/sec execute 295 sec latency 7.741 ms oleg457-client.virtnet: 1 121091 2.00 MB/sec execute 296 sec latency 8.267 ms oleg457-client.virtnet: 1 121777 2.00 MB/sec execute 297 sec latency 11.858 ms oleg457-client.virtnet: 1 122473 2.00 MB/sec execute 298 sec latency 9.077 ms oleg457-client.virtnet: 1 123350 2.02 MB/sec execute 299 sec latency 7.894 ms oleg457-client.virtnet: 1 cleanup 300 sec oleg457-client.virtnet: 0 cleanup 300 sec oleg457-client.virtnet: oleg457-client.virtnet: Operation Count AvgLat MaxLat oleg457-client.virtnet: ---------------------------------------- oleg457-client.virtnet: NTCreateX 19450 9.238 19837.130 oleg457-client.virtnet: Close 14321 2.746 19946.936 oleg457-client.virtnet: Rename 828 9.021 23.260 oleg457-client.virtnet: Unlink 3901 2.375 8.977 oleg457-client.virtnet: Qpathinfo 17676 1.697 18.294 oleg457-client.virtnet: Qfileinfo 3094 0.171 3.104 oleg457-client.virtnet: Qfsinfo 3239 0.285 11.661 oleg457-client.virtnet: Sfileinfo 1582 4.746 15.742 oleg457-client.virtnet: Find 6835 0.487 17.691 oleg457-client.virtnet: WriteX 9646 1.089 13.812 oleg457-client.virtnet: ReadX 30719 0.021 2.221 oleg457-client.virtnet: LockX 64 1.427 2.290 oleg457-client.virtnet: UnlockX 64 1.450 2.123 oleg457-client.virtnet: Flush 1352 7.247 584.647 oleg457-client.virtnet: oleg457-client.virtnet: Throughput 2.01605 MB/sec 1 clients 1 procs max_latency=19946.969 ms oleg457-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg457-client.virtnet at Thu Apr 18 04:27:43 EDT 2024 with return code 0 oleg457-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg457-client.virtnet oleg457-client.virtnet: /mnt/lustre/d70b.replay-single/oleg457-client.virtnet /mnt/lustre/d70b.replay-single/oleg457-client.virtnet oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg457-client.virtnet: removed directory: 'clients/client0' oleg457-client.virtnet: removed directory: 'clients' oleg457-client.virtnet: removed 'client.txt' oleg457-client.virtnet: /mnt/lustre/d70b.replay-single/oleg457-client.virtnet oleg457-client.virtnet: dbench successfully finished PASS 70b (363s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70c: tar 2mdts recovery ============ 04:27:45 (1713428865) Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar 2225 striped dir -i0 -c2 -H crush2 /mnt/lustre/d70c.replay-single tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 6892 1280796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4648 1283040 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13504 3572060 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7840 3582524 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 21344 7154584 1% /mnt/lustre test_70c fail mds1 1 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:29:59 (1713428999) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:30:13 (1713429013) targets are mounted 04:30:13 (1713429013) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 7016 1280672 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4688 1283000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 4940 3583624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18668 3568268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 23608 7151892 1% /mnt/lustre test_70c fail mds1 2 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:32:37 (1713429157) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:32:52 (1713429172) targets are mounted 04:32:52 (1713429172) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70c (341s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 04:33:27 (1713429207) Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started 5573 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4868 1282820 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4584 1283104 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3599564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3596048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7195612 1% /mnt/lustre test_70d fail mds1 1 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:35:47 (1713429347) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:36:01 (1713429361) targets are mounted 04:36:01 (1713429361) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3984 1283704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5364 1282324 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3599564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3596048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7195612 1% /mnt/lustre test_70d fail mds1 2 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:38:27 (1713429507) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:38:43 (1713429523) targets are mounted 04:38:43 (1713429523) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 5573 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test || { echo "mkdir fails"; break; }; $LFS mkdir -i1 -c2 $DIR/$tdir/test1 || { echo "mkdir fails"; break; }; touch $DIR/$tdir/test/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test || { echo "rmdir fails"; ls -lR $DIR/$tdir; break; }; touch $DIR/$tdir/test1/a || { echo "touch fails"; break; }; mkdir $DIR/$tdir/test1/b || { echo "mkdir fails"; break; }; rm -rf $DIR/$tdir/test1 || { echo "rmdir fails"; ls -lR $DIR/$tdir/test1; break; }; done ) (wd: ~) PASS 70d (327s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70e: rename cross-MDT with random fails ========================================================== 04:38:57 (1713429537) debug=+ha Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started PID=31507 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3880 1283808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2868 1284820 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3599564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3596048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7195612 1% /mnt/lustre test_70e fail mds1 1 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:41:13 (1713429673) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:41:29 (1713429689) targets are mounted 04:41:29 (1713429689) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3616 1284072 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2804 1284884 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3599564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3596048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7195612 1% /mnt/lustre test_70e fail mds1 2 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:43:52 (1713429832) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:44:06 (1713429846) targets are mounted 04:44:06 (1713429846) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 70e (321s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 04:44:19 (1713429859) mount clients oleg457-client.virtnet ... Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3420 1284268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3583620 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3593036 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7176656 1% /mnt/lustre ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... test_70f failing OST 1 times Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 04:44:28 (1713429868) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... Started lustre-OST0000 ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear 04:44:43 (1713429883) targets are mounted 04:44:43 (1713429883) facet_failover done Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3420 1284268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3580476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 4624 3601348 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 6208 7181824 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... test_70f failing OST 2 times Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 04:44:56 (1713429896) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Started lustre-OST0000 04:45:10 (1713429910) targets are mounted 04:45:10 (1713429910) facet_failover done Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3420 1284268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 5680 3557708 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3593036 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 7232 7150744 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... test_70f failing OST 3 times Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 04:45:24 (1713429924) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... Started lustre-OST0000 04:45:39 (1713429939) targets are mounted 04:45:39 (1713429939) facet_failover done ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3420 1284268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3586764 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3576412 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7163176 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... test_70f failing OST 4 times Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 04:45:52 (1713429952) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 04:46:07 (1713429967) targets are mounted 04:46:07 (1713429967) facet_failover done ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3420 1284268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3561804 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3593036 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7154840 1% /mnt/lustre Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... test_70f failing OST 5 times Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 04:46:20 (1713429980) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... Started lustre-OST0000 ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear 04:46:35 (1713429995) targets are mounted 04:46:35 (1713429995) facet_failover done Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b42ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b42ec800.lru_size=clear Write/read files in: '/mnt/lustre/d70f.replay-single', clients: 'oleg457-client.virtnet' ... PASS 70f (143s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 04:46:44 (1713430004) Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started 10898 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5980 1281708 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4392 1283296 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4100 1283588 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4560 1283128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre fail mds2 mds1 1 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:49:07 (1713430147) shut down Failover mds1 to oleg457-server Failover mds2 to oleg457-server mount facets: mds2 mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 04:49:21 (1713430161) targets are mounted 04:49:21 (1713430161) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4452 1283236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3872 1283816 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4452 1283236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3872 1283816 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre fail mds2 mds1 2 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:51:52 (1713430312) shut down Failover mds1 to oleg457-server Failover mds2 to oleg457-server mount facets: mds2 mount facets: mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 04:52:06 (1713430326) targets are mounted 04:52:06 (1713430326) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 10898 Killed ( while true; do $LFS mkdir -i0 -c2 $DIR/$tdir/test; rmdir $DIR/$tdir/test; done ) (wd: ~) PASS 71a (335s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 04:52:21 (1713430341) multiop /mnt/lustre/f73a.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4216 1283472 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 6084 1281604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre fail_loc=0x80000302 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:52:24 (1713430344) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:52:38 (1713430358) targets are mounted 04:52:38 (1713430358) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73a (38s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 04:53:01 (1713430381) multiop /mnt/lustre/f73b.replay-single vO_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7497 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2012 1285676 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre fail_loc=0x80000157 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:53:04 (1713430384) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:53:18 (1713430398) targets are mounted 04:53:18 (1713430398) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 73b (38s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 04:53:41 (1713430421) Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:) Stopping client oleg457-client.virtnet /mnt/lustre opts: Stopping /mnt/lustre-ost1 (opts:) on oleg457-server Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:53:44 (1713430424) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:53:57 (1713430437) targets are mounted 04:53:57 (1713430437) facet_failover done Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 PASS 74 (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 04:54:09 (1713430449) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:54:12 (1713430452) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:54:26 (1713430466) targets are mounted 04:54:26 (1713430466) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.04 seconds: 460.27 ops/second PASS 80a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 04:54:35 (1713430475) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:54:46 (1713430486) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:55:00 (1713430500) targets are mounted 04:55:00 (1713430500) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.04 seconds: 486.95 ops/second PASS 80b (32s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 04:55:08 (1713430508) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:55:13 (1713430513) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:55:27 (1713430527) targets are mounted 04:55:27 (1713430527) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:55:34 (1713430534) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:55:48 (1713430548) targets are mounted 04:55:48 (1713430548) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 410.80 ops/second PASS 80c (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 04:55:56 (1713430556) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:56:12 (1713430572) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 04:56:32 (1713430592) targets are mounted 04:56:32 (1713430592) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.06 seconds: 357.95 ops/second PASS 80d (55s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 04:56:52 (1713430612) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:56:59 (1713430619) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:57:13 (1713430633) targets are mounted 04:57:13 (1713430633) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.06 seconds: 358.78 ops/second PASS 80e (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 04:57:22 (1713430642) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:57:26 (1713430646) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:57:40 (1713430660) targets are mounted 04:57:40 (1713430660) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.06 seconds: 331.27 ops/second PASS 80f (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 04:57:48 (1713430668) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:57:57 (1713430677) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:58:11 (1713430691) targets are mounted 04:58:11 (1713430691) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:58:18 (1713430698) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:58:32 (1713430712) targets are mounted 04:58:32 (1713430712) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 442.64 ops/second PASS 80g (51s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 04:58:41 (1713430721) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:58:51 (1713430731) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 04:59:17 (1713430757) targets are mounted 04:59:17 (1713430757) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec total: 20 open/close in 0.05 seconds: 382.52 ops/second PASS 80h (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 04:59:29 (1713430769) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:59:38 (1713430778) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:59:52 (1713430792) targets are mounted 04:59:52 (1713430792) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81a (30s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 05:00:00 (1713430800) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:00:04 (1713430804) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:00:18 (1713430818) targets are mounted 05:00:18 (1713430818) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81b (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 05:00:27 (1713430827) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:00:32 (1713430832) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:00:46 (1713430846) targets are mounted 05:00:46 (1713430846) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:00:53 (1713430853) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:01:07 (1713430867) targets are mounted 05:01:07 (1713430867) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81c (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 05:01:16 (1713430876) fail_loc=0x1701 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:01:28 (1713430888) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 05:01:48 (1713430908) targets are mounted 05:01:48 (1713430908) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81d (40s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 05:01:57 (1713430917) fail_loc=0x119 fail_loc=0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:02:01 (1713430921) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:02:15 (1713430935) targets are mounted 05:02:15 (1713430935) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81e (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 05:02:24 (1713430944) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:02:27 (1713430947) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:02:41 (1713430961) targets are mounted 05:02:41 (1713430961) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81f (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 05:02:50 (1713430970) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:02:55 (1713430975) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:03:09 (1713430989) targets are mounted 05:03:09 (1713430989) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:03:16 (1713430996) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:03:30 (1713431010) targets are mounted 05:03:30 (1713431010) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81g (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 05:03:39 (1713431019) fail_loc=0x119 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3136 7210904 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:03:46 (1713431026) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 05:04:12 (1713431052) targets are mounted 05:04:12 (1713431052) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 81h (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 84a: stale open during export disconnect ========================================================== 05:04:27 (1713431067) fail_loc=0x80000144 total: 1 open/close in 0.01 seconds: 178.32 ops/second pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 PASS 84a (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 05:04:33 (1713431073) before recovery: unused locks count = 201 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:04:36 (1713431076) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:04:49 (1713431089) targets are mounted 05:04:49 (1713431089) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec after recovery: unused locks count = 101 PASS 85a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 05:04:58 (1713431098) before recovery: unused locks count = 100 Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 05:05:02 (1713431102) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 05:05:15 (1713431115) targets are mounted 05:05:15 (1713431115) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec after recovery: unused locks count = 0 PASS 85b (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 05:05:21 (1713431121) Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:) Stopping client oleg457-client.virtnet /mnt/lustre opts: mdt.lustre-MDT0000.exports.clear=0 mdt.lustre-MDT0001.exports.clear=0 Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) PASS 86 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 87a: write replay ================== 05:05:34 (1713431134) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1784 3605236 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1752 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3536 7210504 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.124002 s, 67.6 MB/s Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 05:05:38 (1713431138) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 05:05:52 (1713431152) targets are mounted 05:05:52 (1713431152) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0389361 s, 215 MB/s PASS 87a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 05:05:58 (1713431158) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9976 3597044 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1752 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11728 7202312 1% /mnt/lustre 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.127691 s, 65.7 MB/s 8+0 records in 8+0 records out 8 bytes (8 B) copied, 0.00180774 s, 4.4 kB/s Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 05:06:03 (1713431163) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 05:06:18 (1713431178) targets are mounted 05:06:18 (1713431178) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 0+1 records in 0+1 records out 72 bytes (72 B) copied, 0.00265444 s, 27.1 kB/s PASS 87b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 88: MDS should not assign same objid to different files ========================================================== 05:06:24 (1713431184) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9980 3597040 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1752 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11732 7202308 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9980 3597040 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1752 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 11732 7202308 1% /mnt/lustre before test: last_id = 14273, next_id = 14244 Creating to objid 14273 on ost lustre-OST0000... total: 31 open/close in 0.07 seconds: 427.95 ops/second total: 8 open/close in 0.02 seconds: 450.20 ops/second before recovery: last_id = 14305, next_id = 14282 Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Failover ost1 to oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 after recovery: last_id = 14313, next_id = 14282 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0336882 s, 15.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0273573 s, 19.2 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0279717 s, 18.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0284826 s, 18.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0339355 s, 15.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0270546 s, 19.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0284785 s, 18.4 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0300342 s, 17.5 MB/s -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14244 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14245 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14246 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14247 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14248 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14249 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14250 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14251 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14252 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14253 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14254 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14255 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14256 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14257 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14258 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14259 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14260 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14261 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14262 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14263 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14264 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14265 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14266 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14267 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14268 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14269 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14270 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14271 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14272 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14273 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14274 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14275 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14276 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14277 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14278 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14279 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14280 -rw-r--r-- 1 root root 0 Apr 18 05:06 /mnt/lustre/d88.replay-single/f-14281 -rw-r--r-- 1 root root 524288 Apr 18 05:07 /mnt/lustre/d88.replay-single/f-14285 -rw-r--r-- 1 root root 524288 Apr 18 05:07 /mnt/lustre/d88.replay-single/f-14286 -rw-r--r-- 1 root root 524288 Apr 18 05:07 /mnt/lustre/d88.replay-single/f-14287 -rw-r--r-- 1 root root 524288 Apr 18 05:07 /mnt/lustre/d88.replay-single/f-14288 -rw-r--r-- 1 root root 524288 Apr 18 05:07 /mnt/lustre/d88.replay-single/f-14289 -rw-r--r-- 1 root root 524288 Apr 18 05:07 /mnt/lustre/d88.replay-single/f-14290 -rw-r--r-- 1 root root 524288 Apr 18 05:07 /mnt/lustre/d88.replay-single/f-14291 -rw-r--r-- 1 root root 524288 Apr 18 05:07 /mnt/lustre/d88.replay-single/f-14292 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0265186 s, 19.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0307884 s, 17.0 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0280514 s, 18.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0265393 s, 19.8 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0280757 s, 18.7 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0297435 s, 17.6 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0255423 s, 20.5 MB/s 128+0 records in 128+0 records out 524288 bytes (524 kB) copied, 0.0301389 s, 17.4 MB/s PASS 88 (52s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 89: no disk space leak on late ost connection ========================================================== 05:07:17 (1713431237) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. Waiting for MDT destroys to complete 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.068654 s, 153 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg457-server Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:07:23 (1713431243) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:07:36 (1713431256) targets are mounted 05:07:36 (1713431256) facet_failover done Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. Waiting for MDT destroys to complete free_before: 7646308 free_after: 7646308 PASS 89 (109s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 05:09:07 (1713431347) Create the files Fail ost2 lustre-OST0001_UUID, display the list of affected files Stopping /mnt/lustre-ost2 (opts:) on oleg457-server General Query: lfs find /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single /mnt/lustre/d90.replay-single/f1 /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f0 Querying files on shutdown ost2: lfs find --obd lustre-OST0001_UUID /mnt/lustre/d90.replay-single/f1 /mnt/lustre/d90.replay-single/all Check getstripe: /home/green/git/lustre-release/lustre/utils/lfs getstripe -r --obd lustre-OST0001_UUID /mnt/lustre/d90.replay-single/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 14243 0x37a3 0x2c0000401 * /mnt/lustre/d90.replay-single/all lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 1 14242 0x37a2 0x2c0000401 * /mnt/lustre/d90.replay-single/all /mnt/lustre/d90.replay-single/f1 Failover ost2 to oleg457-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 PASS 90 (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 93a: replay + reconnect ============ 05:09:27 (1713431367) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00178243 s, 574 kB/s fail_val=40 fail_loc=0x715 Failing ost1 on oleg457-server Stopping /mnt/lustre-ost1 (opts:) on oleg457-server 05:09:29 (1713431369) shut down Failover ost1 to oleg457-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 05:09:42 (1713431382) targets are mounted 05:09:42 (1713431382) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 93a (58s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 93b: replay + reconnect on mds ===== 05:10:27 (1713431427) total: 20 open/close in 0.07 seconds: 306.71 ops/second fail_val=80 fail_loc=0x715 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:10:29 (1713431429) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:10:42 (1713431442) targets are mounted 05:10:42 (1713431442) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 93b (102s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 05:12:10 (1713431530) fail_loc=0x1701 Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:12:12 (1713431532) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:12:25 (1713431545) targets are mounted 05:12:25 (1713431545) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200034fc0:0x1:0x0] 1 [0x240009c5a:0x1:0x0] total: 20 open/close in 0.05 seconds: 401.99 ops/second PASS 100a (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 05:12:34 (1713431554) fail_loc=0x119 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:12:35 (1713431555) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:12:48 (1713431568) targets are mounted 05:12:48 (1713431568) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200034fc0:0x4:0x0] 1 [0x240009c5a:0x4:0x0] total: 20 open/close in 0.05 seconds: 425.19 ops/second PASS 100b (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 05:12:57 (1713431577) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2536 1285152 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2140 1285548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Stopping /mnt/lustre-mds2 (opts:) on oleg457-server Failover mds2 to oleg457-server Starting mds2: -o localrecov -o abort_recov_mdt /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 total: 20 open/close in 0.06 seconds: 328.34 ops/second Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:13:17 (1713431597) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:13:30 (1713431610) targets are mounted 05:13:30 (1713431610) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 1 [0x24000a428:0x3:0x0] 0 [0x200034fc1:0x3:0x0] total: 20 open/close in 0.04 seconds: 475.77 ops/second PASS 100c (40s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 05:13:39 (1713431619) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d100d.replay-single total: 100 mkdir in 0.25 seconds: 405.03 ops/second lustre-MDT0000-osd [catalog]: [0x20001a210:0x1:0x0] [index]: 00026 [logid]: [0x200035790:0x1:0x0] lustre-MDT0001-osp-MDT0000 [catalog]: [0x2400007ec:0x1:0x0] [index]: 00037 [logid]: [0x24000a429:0x1:0x0] [index]: 00038 [logid]: [0x24000a429:0x2:0x0] Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 first stat failed: 5 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 find: '/mnt/lustre/d100d.replay-single': No such file or directory find: '/mnt/lustre/d100d.replay-single': No such file or directory PASS 100d (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 05:13:59 (1713431639) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2956 1284732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2556 1285132 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2956 1284732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2556 1285132 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200035f60:0x2:0x0] 1 [0x24000b3c9:0x2:0x0] Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:14:05 (1713431645) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 05:14:31 (1713431671) targets are mounted 05:14:31 (1713431671) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200035f60:0x2:0x0] 1 [0x24000b3c9:0x2:0x0] PASS 100e (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 05:14:41 (1713431681) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3072 1284616 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2660 1285028 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 first stat failed: 5 PASS 101 (34s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 05:15:17 (1713431717) creating 7 files ... fail_loc=0x159 launch 7 chmod in parallel (05:15:17) ... fail_loc=0 done (05:15:33) /mnt/lustre/d102a.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-7 has perms 0600 OK PASS 102a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 05:15:36 (1713431736) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (05:15:37) ... fail_loc=0 done (05:15:53) /mnt/lustre/d102b.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-7 has perms 0600 OK PASS 102b (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 05:15:56 (1713431756) creating 7 files ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2944 1284744 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3580640 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3597104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7177744 1% /mnt/lustre fail_loc=0x15a launch 7 chmod in parallel (05:15:58) ... fail_loc=0 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:16:01 (1713431761) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:16:14 (1713431774) targets are mounted 05:16:14 (1713431774) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec done (05:16:20) /mnt/lustre/d102c.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-7 has perms 0600 OK PASS 102c (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight ========================================================== 05:16:23 (1713431783) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (05:16:24) ... fail_loc=0 Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:16:26 (1713431786) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:16:39 (1713431799) targets are mounted 05:16:39 (1713431799) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec done (05:16:45) /mnt/lustre/d102d.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-7 has perms 0600 OK PASS 102d (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 103: Check otr_next_id overflow ==== 05:16:48 (1713431808) fail_loc=0x80000162 total: 30 open/close in 0.07 seconds: 410.32 ops/second Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:16:50 (1713431810) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:17:03 (1713431823) targets are mounted 05:17:03 (1713431823) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 103 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 05:17:11 (1713431831) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2896 1284792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3580640 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3597104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7177744 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:17:15 (1713431835) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:17:29 (1713431849) targets are mounted 05:17:29 (1713431849) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110a.replay-single/striped_dir has type dir OK PASS 110a (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 05:17:37 (1713431857) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2896 1284792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3580640 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3597104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7177744 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:17:41 (1713431861) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:17:55 (1713431875) targets are mounted 05:17:55 (1713431875) facet_failover done pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre /mnt/lustre/d110b.replay-single/striped_dir has type dir OK PASS 110b (93s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 05:19:12 (1713431952) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2904 1284784 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:19:15 (1713431955) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:19:29 (1713431969) targets are mounted 05:19:29 (1713431969) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110c.replay-single/striped_dir has type dir OK PASS 110c (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 05:19:38 (1713431978) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2944 1284744 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:19:41 (1713431981) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:19:55 (1713431995) targets are mounted 05:19:55 (1713431995) facet_failover done pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre /mnt/lustre/d110d.replay-single/striped_dir has type dir OK PASS 110d (92s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 05:21:11 (1713432071) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2980 1284708 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2508 1285180 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:21:18 (1713432078) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 05:21:44 (1713432104) targets are mounted 05:21:44 (1713432104) facet_failover done pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre /mnt/lustre/d110e.replay-single/striped_dir has type dir OK PASS 110e (107s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-single test_110f skipping excluded test 110f debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 05:23:00 (1713432180) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3024 1284664 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:23:12 (1713432192) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 05:23:33 (1713432213) targets are mounted 05:23:33 (1713432213) facet_failover done pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre /mnt/lustre/d110g.replay-single/striped_dir has type dir OK PASS 110g (109s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 05:24:50 (1713432290) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3072 1284616 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:24:53 (1713432293) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:25:07 (1713432307) targets are mounted 05:25:07 (1713432307) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111a.replay-single/striped_dir: No such file or directory PASS 111a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 05:25:16 (1713432316) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2900 1284788 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:25:19 (1713432319) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:25:33 (1713432333) targets are mounted 05:25:33 (1713432333) facet_failover done pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111b.replay-single/striped_dir: No such file or directory PASS 111b (91s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 05:26:49 (1713432409) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2908 1284780 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:27:01 (1713432421) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 05:27:21 (1713432441) targets are mounted 05:27:21 (1713432441) facet_failover done pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111c.replay-single/striped_dir: No such file or directory PASS 111c (108s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 05:28:38 (1713432518) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2984 1284704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:28:45 (1713432525) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 05:29:11 (1713432551) targets are mounted 05:29:11 (1713432551) facet_failover done pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg457-client: oleg457-client: ssh exited with exit code 95 Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111d.replay-single/striped_dir: No such file or directory PASS 111d (107s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 05:30:27 (1713432627) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2912 1284776 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2952 1284736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:30:34 (1713432634) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 05:30:59 (1713432659) targets are mounted 05:30:59 (1713432659) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111e.replay-single/striped_dir: No such file or directory PASS 111e (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 05:31:09 (1713432669) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2920 1284768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2920 1284768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:31:16 (1713432676) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 05:31:41 (1713432701) targets are mounted 05:31:41 (1713432701) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111f.replay-single/striped_dir: No such file or directory PASS 111f (40s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 05:31:51 (1713432711) UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 552 1023448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 322 1023678 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 557 261587 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 487 261657 1% /mnt/lustre[OST:1] filesystem_summary: 524118 874 523244 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2916 1284772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2928 1284760 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:31:58 (1713432718) shut down Failover mds1 to oleg457-server mount facets: mds1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 05:32:23 (1713432743) targets are mounted 05:32:23 (1713432743) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111g.replay-single/striped_dir: No such file or directory PASS 111g (39s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 05:32:32 (1713432752) SKIP: replay-single test_112a needs >= 4 MDTs SKIP 112a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 05:32:34 (1713432754) SKIP: replay-single test_112b needs >= 4 MDTs SKIP 112b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 05:32:37 (1713432757) SKIP: replay-single test_112c needs >= 4 MDTs SKIP 112c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 05:32:40 (1713432760) SKIP: replay-single test_112d needs >= 4 MDTs SKIP 112d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 05:32:43 (1713432763) SKIP: replay-single test_112e needs >= 4 MDTs SKIP 112e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 05:32:46 (1713432766) SKIP: replay-single test_112f needs >= 4 MDTs SKIP 112f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 05:32:49 (1713432769) SKIP: replay-single test_112g needs >= 4 MDTs SKIP 112g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 05:32:52 (1713432772) SKIP: replay-single test_112h needs >= 4 MDTs SKIP 112h (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 05:32:55 (1713432775) SKIP: replay-single test_112i needs >= 4 MDTs SKIP 112i (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 05:32:57 (1713432777) SKIP: replay-single test_112j needs >= 4 MDTs SKIP 112j (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 05:33:00 (1713432780) SKIP: replay-single test_112k needs >= 4 MDTs SKIP 112k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 05:33:04 (1713432784) SKIP: replay-single test_112l needs >= 4 MDTs SKIP 112l (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 05:33:07 (1713432787) SKIP: replay-single test_112m needs >= 4 MDTs SKIP 112m (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 05:33:10 (1713432790) SKIP: replay-single test_112n needs >= 4 MDTs SKIP 112n (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 115: failover for create/unlink striped directory ========================================================== 05:33:13 (1713432793) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2924 1284764 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_2 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_3 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_4 Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:33:18 (1713432798) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:33:32 (1713432812) targets are mounted 05:33:32 (1713432812) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2976 1284712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre striped dir -i0 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_0 striped dir -i0 -c2 -H crush /mnt/lustre/d115.replay-single/test_1 striped dir -i0 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_2 striped dir -i0 -c2 -H crush2 /mnt/lustre/d115.replay-single/test_3 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d115.replay-single/test_4 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:33:43 (1713432823) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:33:58 (1713432838) targets are mounted 05:33:58 (1713432838) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 115 (53s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 116a: large update log master MDT recovery ========================================================== 05:34:08 (1713432848) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3016 1284672 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2608 1285080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre fail_loc=0x80001702 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:34:12 (1713432852) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:34:26 (1713432866) targets are mounted 05:34:26 (1713432866) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d116a.replay-single/striped_dir has type dir OK PASS 116a (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 116b: large update log slave MDT recovery ========================================================== 05:34:35 (1713432875) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3068 1284620 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2780 1284908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre fail_loc=0x80001702 Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 05:34:40 (1713432880) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 05:34:54 (1713432894) targets are mounted 05:34:54 (1713432894) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d116b.replay-single/striped_dir has type dir OK PASS 116b (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 05:35:03 (1713432903) SKIP: replay-single test_117 needs >= 4 MDTs SKIP 117 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 05:35:06 (1713432906) fail_loc=0x1705 lfs setdirstripe: dirstripe error on '/mnt/lustre/d118.replay-single/striped_dir': Input/output error lfs setdirstripe: cannot create dir '/mnt/lustre/d118.replay-single/striped_dir': Input/output error lfs setdirstripe: dirstripe error on '/mnt/lustre/d118.replay-single/striped_dir1': Input/output error lfs setdirstripe: cannot create dir '/mnt/lustre/d118.replay-single/striped_dir1': Input/output error fail_loc=0x0 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3164 1284524 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2660 1285028 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:35:11 (1713432911) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:26 (1713432926) targets are mounted 05:35:26 (1713432926) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 118 (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 05:35:35 (1713432935) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3232 1284456 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2720 1284968 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server fail_loc=0x80000714 fail_val=65 Starting mds1: -o localrecov -o recovery_time_hard=60 /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 71 sec mdt.lustre-MDT0000.recovery_time_hard=180 PASS 119 (88s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 05:37:05 (1713433025) Replay barrier on lustre-MDT0000 Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server Starting mds1: -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 first stat failed: 5 find: '/mnt/lustre/d120.replay-single': No such file or directory PASS 120 (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 121: lock replay timed out and race ========================================================== 05:37:35 (1713433055) multiop /mnt/lustre/f121.replay-single vs_s TMPPIPE=/tmp/multiop_open_wait_pipe.7497 Stopping /mnt/lustre-mds1 (opts:) on oleg457-server Failover mds1 to oleg457-server fail_loc=0x721 fail_val=0 at_max=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 fail_loc=0x0 at_max=600 PASS 121 (200s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 05:40:58 (1713433258) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3632 1284056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3024 1284664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:41:03 (1713433263) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:41:18 (1713433278) targets are mounted 05:41:18 (1713433278) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130a (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 130b: DoM file create (inherited) replay ========================================================== 05:41:29 (1713433289) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3636 1284052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3024 1284664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:41:34 (1713433294) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:41:49 (1713433309) targets are mounted 05:41:49 (1713433309) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 130b (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 131a: DoM file write lock replay === 05:42:00 (1713433320) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3636 1284052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3024 1284664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre 1+0 records in 1+0 records out 8 bytes (8 B) copied, 0.00223955 s, 3.6 kB/s Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:42:05 (1713433325) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:42:20 (1713433340) targets are mounted 05:42:20 (1713433340) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 131a (28s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-single test_131b skipping excluded test 131b debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 132a: PFL new component instantiate replay ========================================================== 05:42:31 (1713433351) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3640 1284048 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3024 1284664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1768 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19944 7194096 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350825 s, 29.9 MB/s /mnt/lustre/f132a.replay-single lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f2a:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3ee2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x3f2b:0x0] } Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:42:36 (1713433356) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:42:52 (1713433372) targets are mounted 05:42:52 (1713433372) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/f132a.replay-single lcm_layout_gen: 1 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f2a:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 65535 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3ee2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x3f2b:0x0] } PASS 132a (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 05:43:03 (1713433383) seq.srv-lustre-MDT0001.space=clear Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre fail_val=700 fail_loc=0x80000123 Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:43:08 (1713433388) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:43:22 (1713433402) targets are mounted 05:43:22 (1713433402) facet_failover done PASS 133 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 134: replay creation of a file created in a pool ========================================================== 05:43:30 (1713433410) Creating new pool oleg457-server: Pool lustre.pool_134 created Adding targets to pool oleg457-server: OST lustre-OST0001_UUID added to pool lustre.pool_134 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3724 1283964 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3068 1284620 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2792 3604228 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 05:43:41 (1713433421) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 05:43:57 (1713433437) targets are mounted 05:43:57 (1713433437) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Destroy the created pools: pool_134 lustre.pool_134 oleg457-server: OST lustre-OST0001_UUID removed from pool lustre.pool_134 oleg457-server: Pool lustre.pool_134 destroyed PASS 134 (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 135: Server failure in lock replay phase ========================================================== 05:44:13 (1713433453) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3704 1283984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3068 1284620 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18176 3588844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2792 3604228 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 20968 7193072 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 Stopping /mnt/lustre-ost1 (opts:) on oleg457-server Failover ost1 to oleg457-server oleg457-server: oleg457-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg457-server Failover ost1 to oleg457-server oleg457-server: oleg457-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all End of sync pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 5 ldlm.cancel_unused_locks_before_replay=1 PASS 135 (77s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 05:45:32 (1713433532) SKIP: replay-single test_136 needs > 2 MDTs SKIP 136 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 05:45:35 (1713433535) SKIP: replay-single test_200 Need remote client SKIP 200 (2s) debug_raw_pointers=0 debug_raw_pointers=0 == replay-single test complete, duration 8528 sec ======== 05:45:37 (1713433537) === replay-single: start cleanup 05:45:38 (1713433538) === === replay-single: finish cleanup 05:45:41 (1713433541) ===