[ 1256.280801] Lustre: Failing over lustre-MDT0000
[ 1256.396941] Lustre: server umount lustre-MDT0000 complete
[ 1259.233292] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1259.241750] LustreError: Skipped 1 previous similar message
[ 1259.424421] Lustre: lustre-MDT0000: Aborting client recovery
[ 1259.426700] LustreError: 13708:0:(ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1259.430131] Lustre: 13849:0:(ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1259.435352] Lustre: 13849:0:(ldlm_lib.c:2310:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1259.439591] Lustre: 13849:0:(genops.c:1516:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0@<unknown>
[ 1259.445681] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 1259.464394] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 1259.517005] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:1603 to 0x280000401:1665
[ 1259.517052] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:1576 to 0x240000401:1665
[ 1260.603208] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1268.331370] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:09:31 (1713532171)
[ 1278.535508] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1279.262251] Lustre: Failing over lustre-MDT0000
[ 1279.407046] Lustre: server umount lustre-MDT0000 complete
[ 1292.115260] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1292.118041] Lustre: Skipped 22 previous similar messages
[ 1293.216218] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1295.312103] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:2066 to 0x280000401:2081
[ 1295.312114] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:2066 to 0x240000401:2081
[ 1296.150803] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1296.719658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1307.536643] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:10:10 (1713532210)
[ 1314.385015] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1318.006873] Lustre: Failing over lustre-MDT0000
[ 1318.140148] Lustre: server umount lustre-MDT0000 complete
[ 1330.785890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1330.789508] Lustre: Skipped 22 previous similar messages
[ 1331.691650] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1331.988128] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:2482 to 0x280000401:2497
[ 1331.989583] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:2482 to 0x240000401:2497
[ 1334.664353] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1335.252091] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1346.688622] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 09:10:49 (1713532249)
[ 1347.489634] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 1347.872074] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1347.877465] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 1352.022372] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 09:10:55 (1713532255)
[ 1359.006426] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 1362.891238] Lustre: Failing over lustre-OST0000
[ 1362.938756] Lustre: server umount lustre-OST0000 complete
[ 1365.808896] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 1365.814056] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1370.832844] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1370.840437] LustreError: Skipped 1 previous similar message
[ 1377.129882] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1425.469982] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 09:12:08 (1713532328)
[ 1426.929586] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1427.936083] Lustre: Failing over lustre-MDT0000
[ 1428.080343] Lustre: server umount lustre-MDT0000 complete
[ 1440.436081] LustreError: 166-1: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1440.443017] LustreError: Skipped 18 previous similar messages
[ 1440.480842] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1440.486907] Lustre: Skipped 42 previous similar messages
[ 1440.487697] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1440.487699] LustreError: Skipped 1 previous similar message
[ 1441.331018] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1441.335531] Lustre: Skipped 16 previous similar messages
[ 1441.354170] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1441.357434] Lustre: Skipped 16 previous similar messages
[ 1441.374345] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 1441.374363] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:2899 to 0x240000401:2945
[ 1441.594703] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1444.384574] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1444.954456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1445.611520] Lustre: 2637:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713532334/real 1713532334]  req@ffff88008e734000 x1796767410056320/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1713532350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 1445.618148] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to  (at 0@lo)
[ 1445.618150] Lustre: Skipped 43 previous similar messages
[ 1445.629220] Lustre: 2637:0:(client.c:2340:ptlrpc_expire_one_request()) Skipped 88 previous similar messages
[ 1457.373493] LustreError: 29225:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 1457.373976] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1458.382867] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:2898 to 0x280000401:2913
[ 1460.722436] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 09:12:43 (1713532363)
[ 1462.511628] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_race id 701 sleeping
[ 1467.516397] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1467.521122] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 1467.537362] LustreError: 21770:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_fail_race id 701 waking
[ 1468.085497] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_race id 701 sleeping
[ 1473.089419] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1473.094024] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 1473.604043] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_race id 701 sleeping
[ 1478.608404] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1478.611628] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 1479.190842] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_race id 701 sleeping
[ 1484.194471] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1484.684899] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_race id 701 sleeping
[ 1489.686456] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1489.691536] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 1489.696070] Lustre: Skipped 1 previous similar message
[ 1495.848973] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_race id 701 sleeping
[ 1495.856098] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) Skipped 1 previous similar message
[ 1500.864386] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1500.866282] LustreError: 29187:0:(ldlm_lib.c:1106:target_handle_connect()) Skipped 1 previous similar message
[ 1506.341494] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 1506.346191] Lustre: Skipped 2 previous similar messages
[ 1512.514921] LustreError: 29188:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_race id 701 sleeping
[ 1512.519403] LustreError: 29188:0:(ldlm_lib.c:1106:target_handle_connect()) Skipped 2 previous similar messages
[ 1517.523370] LustreError: 29188:0:(ldlm_lib.c:1106:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1517.525759] LustreError: 29188:0:(ldlm_lib.c:1106:target_handle_connect()) Skipped 2 previous similar messages
[ 1521.488669] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 09:13:44 (1713532424)
[ 1521.955461] LustreError: 29188:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1531.956730] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1536.966511] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1541.974058] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1546.982247] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1551.989855] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1561.957387] LustreError: 29188:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1561.965882] Lustre: 29188:0:(service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff88009cd0ed80 x1796767406092928/t0(0) o38->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1713532446 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0
[ 1562.005990] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 1562.012922] Lustre: Skipped 3 previous similar messages
[ 1562.711845] LustreError: 29461:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1572.712465] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1572.717026] Lustre: Skipped 1 previous similar message
[ 1592.741990] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1592.746795] Lustre: Skipped 4 previous similar messages
[ 1602.716480] LustreError: 29461:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1602.720768] Lustre: 29461:0:(service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff88013617d180 x1796767406094592/t0(0) o38->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1713532487 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0
[ 1602.757679] LustreError: 29461:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1627.758501] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1627.764243] Lustre: Skipped 1 previous similar message
[ 1642.760493] LustreError: 29461:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1642.765022] Lustre: 29461:0:(service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8800a6c8ce00 x1796767406095744/t0(0) o38->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1713532527 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 1642.790093] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 1642.795006] Lustre: Skipped 2 previous similar messages
[ 1642.797353] LustreError: 29187:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1682.802366] LustreError: 29187:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1682.804964] Lustre: 29187:0:(service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff88009396b100 x1796767406096640/t0(0) o38->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1713532567 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 1682.821694] LustreError: 29187:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1707.822322] Lustre: lustre-MDT0000: Export ffff88012d66e800 already connecting from 192.168.201.19@tcp
[ 1707.826820] Lustre: Skipped 5 previous similar messages
[ 1722.826339] LustreError: 29187:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1722.828834] Lustre: 29187:0:(service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8800a6900e00 x1796767406097536/t0(0) o38->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1713532607 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 1722.853397] LustreError: 1754:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1748.555384] LustreError: 1754:0:(ldlm_lib.c:1359:target_handle_connect()) cfs_fail_timeout interrupted
[ 1748.561041] Lustre: 1754:0:(service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/6s); client may timeout  req@ffff8800a8da5f80 x1796767406098432/t0(0) o38->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1713532647 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 1752.405957] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 09:17:35 (1713532655)
[ 1753.559580] LustreError: 3085:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1753.565481] LustreError: 3085:0:(osd_handler.c:698:osd_ro()) Skipped 18 previous similar messages
[ 1753.833628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1755.032880] Lustre: Failing over lustre-MDT0000
[ 1755.153340] Lustre: server umount lustre-MDT0000 complete
[ 1757.857179] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0xbcc0765484121f1b to 0xbcc0765484122e02
[ 1757.880014] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1757.893344] LustreError: Skipped 1 previous similar message
[ 1757.920164] LustreError: 11-0: MGC192.168.201.119@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107
[ 1757.986887] Lustre: *** cfs_fail_loc=712, val=0***
[ 1757.989974] LustreError: 16087:0:(service.c:1236:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff88009437f480 x1796767410101440/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0
[ 1758.059656] Lustre: lustre-MDT0000: Aborting client recovery
[ 1758.065310] LustreError: 4098:0:(ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1758.073942] Lustre: 4221:0:(ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1758.077926] Lustre: 4221:0:(ldlm_lib.c:2310:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1758.081463] Lustre: 4221:0:(genops.c:1516:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0@<unknown>
[ 1758.086506] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 1758.099073] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0]
[ 1758.136790] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:2898 to 0x280000401:2945
[ 1758.136793] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:2899 to 0x240000401:2977
[ 1759.082256] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1762.775566] Lustre: Failing over lustre-MDT0000
[ 1762.904971] Lustre: server umount lustre-MDT0000 complete
[ 1775.716755] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1777.496915] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:2898 to 0x280000401:2977
[ 1777.496917] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:2899 to 0x240000401:3009
[ 1778.077890] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1778.455854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1783.332508] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 09:18:06 (1713532686)
[ 1783.354408] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 1783.360004] Lustre: Skipped 4 previous similar messages
[ 1787.759733] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 09:18:10 (1713532690)
[ 1788.057358] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 1788.060586] LustreError: 5624:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88012f615180 x1796767406123840/t0(0) o700->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:558/0 lens 264/248 e 0 to 0 dl 1713532703 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0
[ 1804.845303] Lustre: Failing over lustre-MDT0000
[ 1804.947803] Lustre: server umount lustre-MDT0000 complete
[ 1816.894984] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1816.897952] Lustre: Skipped 7 previous similar messages
[ 1817.714841] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1818.135225] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:2979 to 0x280000401:3009
[ 1818.135227] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3011 to 0x240000401:3041
[ 1820.240622] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1820.636200] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1826.828638] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 09:18:49 (1713532729)
[ 1827.778727] Lustre: Failing over lustre-OST0000
[ 1827.797424] Lustre: server umount lustre-OST0000 complete
[ 1828.103942] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1831.889098] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 1841.248354] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1843.609641] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 1843.964582] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 1911.449492] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 09:20:14 (1713532814)
[ 1912.887494] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1913.703785] Lustre: Failing over lustre-MDT0000
[ 1913.825067] Lustre: server umount lustre-MDT0000 complete
[ 1926.781088] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1928.382168] Lustre: *** cfs_fail_loc=216, val=0***
[ 1928.382444] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3030 to 0x280000401:3073
[ 1928.388189] LustreError: 13835:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 1929.391689] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3062 to 0x240000401:3105
[ 1992.732640] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 09:21:35 (1713532895)
[ 2002.689231] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 09:21:45 (1713532905)
[ 2003.477878] Lustre: Failing over lustre-MDT0000
[ 2003.574145] Lustre: server umount lustre-MDT0000 complete
[ 2015.996414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2016.001074] Lustre: Skipped 7 previous similar messages
[ 2017.142908] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2018.411981] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 2018.414630] LustreError: 17349:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88008eb7b800 x1796767406179840/t0(0) o101->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:33/0 lens 328/344 e 0 to 0 dl 1713532933 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0
[ 2034.412116] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnected, waiting for 1 clients in recovery for 0:53
[ 2034.448849] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3084 to 0x280000401:3105
[ 2034.453669] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3117 to 0x240000401:3137
[ 2035.102617] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2035.635359] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2041.381119] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 09:22:24 (1713532944)
[ 2042.787630] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2044.222346] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2044.820827] Lustre: Failing over lustre-MDT0000
[ 2044.917310] Lustre: server umount lustre-MDT0000 complete
[ 2057.268157] LustreError: 166-1: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2057.275711] LustreError: Skipped 6 previous similar messages
[ 2057.430945] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2057.439664] Lustre: Skipped 13 previous similar messages
[ 2058.492873] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2058.497281] Lustre: Skipped 5 previous similar messages
[ 2058.520871] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2058.525668] Lustre: Skipped 5 previous similar messages
[ 2058.547253] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3107 to 0x280000401:3137
[ 2058.547255] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3117 to 0x240000401:3169
[ 2058.602471] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2061.494004] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2062.064812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2062.434411] Lustre: 2636:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713532950/real 1713532950]  req@ffff880094200380 x1796767410153408/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1713532966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 2062.449062] Lustre: 2636:0:(client.c:2340:ptlrpc_expire_one_request()) Skipped 25 previous similar messages
[ 2062.450684] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2062.450687] Lustre: Skipped 14 previous similar messages
[ 2067.791329] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 09:22:50 (1713532970)
[ 2068.215554] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2070.464414] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2070.988637] Lustre: Failing over lustre-MDT0000
[ 2071.082215] Lustre: server umount lustre-MDT0000 complete
[ 2083.452520] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2083.530357] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3117 to 0x240000401:3201
[ 2083.530358] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3139 to 0x280000401:3169
[ 2085.678630] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2086.034182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2091.757572] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 09:23:14 (1713532994)
[ 2092.093580] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2094.484466] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2095.173805] Lustre: Failing over lustre-MDT0000
[ 2095.323769] Lustre: server umount lustre-MDT0000 complete
[ 2108.125044] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3139 to 0x280000401:3201
[ 2108.128751] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3203 to 0x240000401:3233
[ 2109.085859] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2116.505815] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 09:23:39 (1713533019)
[ 2117.808005] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2117.809205] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2117.810558] LustreError: 25668:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff880093e95500 x1796767406200768/t257698037777(0) o35->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:133/0 lens 392/456 e 0 to 0 dl 1713533033 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2118.513735] Lustre: Failing over lustre-MDT0000
[ 2118.641097] Lustre: server umount lustre-MDT0000 complete
[ 2131.499430] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2133.156081] Lustre: 28086:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88012ef08a80 x1796767406200768/t257698037777(0) o35->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:148/0 lens 392/456 e 0 to 0 dl 1713533048 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2133.163858] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3139 to 0x280000401:3233
[ 2133.163861] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3235 to 0x240000401:3265
[ 2133.922481] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2134.322480] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2138.767570] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 09:24:01 (1713533041)
[ 2139.132727] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2139.133936] LustreError: 28083:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a097b480 x1796767406207104/t261993005072(0) o36->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:154/0 lens 504/448 e 0 to 0 dl 1713533054 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2141.503348] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2142.013938] Lustre: Failing over lustre-MDT0000
[ 2142.122579] Lustre: server umount lustre-MDT0000 complete
[ 2155.011856] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2155.153184] Lustre: 30861:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8800941c1500 x1796767406207104/t261993005072(0) o36->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:170/0 lens 504/2880 e 0 to 0 dl 1713533070 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2155.167025] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3139 to 0x280000401:3265
[ 2155.170745] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3267 to 0x240000401:3297
[ 2157.496192] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2158.075364] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2163.433068] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 09:24:26 (1713533066)
[ 2163.706952] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2163.709436] LustreError: 30861:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a6b82680 x1796767406213120/t266287972368(0) o36->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:179/0 lens 504/448 e 0 to 0 dl 1713533079 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2165.005747] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2166.149717] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2166.765613] Lustre: Failing over lustre-MDT0000
[ 2166.884963] Lustre: server umount lustre-MDT0000 complete
[ 2179.699742] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2179.720122] Lustre: 1087:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff880129b2f480 x1796767406213248/t266287972369(0) o35->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:195/0 lens 392/456 e 0 to 0 dl 1713533095 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2179.730254] Lustre: 1087:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 2179.731080] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3267 to 0x240000401:3329
[ 2179.731238] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3267 to 0x280000401:3297
[ 2186.031735] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 09:24:49 (1713533089)
[ 2186.334237] LustreError: 1119:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88012f3ef100 x1796767406218944/t270582939664(0) o36->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:201/0 lens 504/448 e 0 to 0 dl 1713533101 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2186.341854] LustreError: 1119:0:(ldlm_lib.c:3271:target_send_reply_msg()) Skipped 1 previous similar message
[ 2187.616323] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2187.617794] Lustre: Skipped 1 previous similar message
[ 2188.955115] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2189.478418] Lustre: Failing over lustre-MDT0000
[ 2189.582255] Lustre: server umount lustre-MDT0000 complete
[ 2202.132657] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2202.353620] Lustre: 3864:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88008e821180 x1796767406218944/t270582939664(0) o36->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:217/0 lens 504/2880 e 0 to 0 dl 1713533117 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2202.368514] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3331 to 0x240000401:3361
[ 2202.368720] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3267 to 0x280000401:3329
[ 2208.193119] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 09:25:11 (1713533111)
[ 2208.497891] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2209.782548] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2209.784459] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2209.786049] Lustre: Skipped 2 previous similar messages
[ 2211.843417] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2212.339317] Lustre: Failing over lustre-MDT0000
[ 2212.360566] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping)
[ 2212.444593] Lustre: server umount lustre-MDT0000 complete
[ 2225.075677] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2232.402226] Lustre: 6388:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8800a6b80a80 x1796767406224512/t274877906960(0) o35->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:247/0 lens 392/456 e 0 to 0 dl 1713533147 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2232.411714] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3331 to 0x240000401:3393
[ 2232.412641] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3331 to 0x280000401:3361
[ 2238.700596] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 09:25:41 (1713533141)
[ 2239.036072] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 2239.039437] LustreError: 6384:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88008e736a00 x1796767406229376/t279172874255(0) o101->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:254/0 lens 664/608 e 0 to 0 dl 1713533154 ref 1 fl Interpret:/200/0 rc 301/0 job:'touch.0' uid:0 gid:0
[ 2239.050285] LustreError: 6384:0:(ldlm_lib.c:3271:target_send_reply_msg()) Skipped 1 previous similar message
[ 2255.034930] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnecting
[ 2255.040140] Lustre: Skipped 4 previous similar messages
[ 2255.044034] Lustre: 6384:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8801357afb80 x1796767406229376/t279172874255(0) o101->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:270/0 lens 664/3488 e 0 to 0 dl 1713533170 ref 1 fl Interpret:/202/0 rc 0/0 job:'touch.0' uid:0 gid:0
[ 2258.635092] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 09:26:01 (1713533161)
[ 2259.694422] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2260.348032] Lustre: Failing over lustre-MDT0000
[ 2260.456780] Lustre: server umount lustre-MDT0000 complete
[ 2273.533680] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2275.118040] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3363 to 0x280000401:3393
[ 2275.118044] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3331 to 0x240000401:3425
[ 2276.021976] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2276.497227] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2291.466324] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 09:26:34 (1713533194)
[ 2292.926896] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2293.458554] Lustre: Failing over lustre-MDT0000
[ 2293.543061] Lustre: server umount lustre-MDT0000 complete
[ 2306.896529] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2308.829644] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:3395 to 0x280000401:3425
[ 2308.830435] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3331 to 0x240000401:3457
[ 2309.364741] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2309.794100] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2311.666772] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2318.431509] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 09:27:01 (1713533221)
[ 2328.377309] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2328.990925] Lustre: Failing over lustre-MDT0000
[ 2329.148704] Lustre: server umount lustre-MDT0000 complete
[ 2342.415269] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2344.332558] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:4676 to 0x280000401:4705
[ 2344.332578] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:4708 to 0x240000401:4737
[ 2344.923067] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2345.434351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2368.279451] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 09:27:51 (1713533271)
[ 2369.303189] LustreError: 18821:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2369.306510] LustreError: 18821:0:(osd_handler.c:698:osd_ro()) Skipped 11 previous similar messages
[ 2369.609422] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2370.314553] Lustre: Failing over lustre-MDT0000
[ 2370.429090] Lustre: server umount lustre-MDT0000 complete
[ 2383.242198] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2383.448721] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:4739 to 0x240000401:4769
[ 2383.448729] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:4676 to 0x280000401:4737
[ 2385.743258] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2386.218993] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2388.518685] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 2389.012213] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 2392.705012] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 09:28:15 (1713533295)
[ 2398.347491] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 2415.072953] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2415.075782] Lustre: Skipped 1 previous similar message
[ 2415.078019] LustreError: 19808:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff8800947f5450 x1796767407698048/t296352743434(0) o36->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:430/0 lens 66040/440 e 0 to 0 dl 1713533330 ref 1 fl Interpret:/200/0 rc 0/0 job:'setfattr.0' uid:0 gid:0
[ 2431.070434] Lustre: 19809:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff880135edd880 x1796767407698048/t296352743434(0) o36->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:446/0 lens 66040/440 e 0 to 0 dl 1713533346 ref 1 fl Interpret:/202/0 rc 0/0 job:'setfattr.0' uid:0 gid:0
[ 2434.229435] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 2436.229337] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 09:28:59 (1713533339)
[ 2439.699929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2440.694785] Lustre: Failing over lustre-MDT0000
[ 2440.851161] Lustre: server umount lustre-MDT0000 complete
[ 2453.218339] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2453.220779] Lustre: Skipped 15 previous similar messages
[ 2454.161681] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2456.354219] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:4871 to 0x240000401:4897
[ 2456.355165] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:4838 to 0x280000401:4865
[ 2457.106009] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2457.611082] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2463.173240] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 09:29:26 (1713533366)
[ 2467.891420] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 2471.159679] Lustre: Failing over lustre-OST0000
[ 2471.182371] Lustre: server umount lustre-OST0000 complete
[ 2471.351996] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107
[ 2471.353164] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2471.353166] LustreError: Skipped 4 previous similar messages
[ 2471.366108] LustreError: Skipped 3 previous similar messages
[ 2476.151681] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2476.156661] LustreError: Skipped 1 previous similar message
[ 2484.476064] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2495.510865] Lustre: Failing over lustre-OST0000
[ 2495.520882] Lustre: server umount lustre-OST0000 complete
[ 2496.183033] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2496.188249] LustreError: Skipped 2 previous similar messages
[ 2508.718416] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2511.052640] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2511.515239] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2547.081212] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 09:30:50 (1713533450)
[ 2547.973009] Lustre: Failing over lustre-MDT0000
[ 2548.108163] Lustre: server umount lustre-MDT0000 complete
[ 2561.238937] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2561.319075] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5298 to 0x240000401:5313
[ 2561.319077] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5266 to 0x280000401:5281
[ 2572.489249] Lustre: Failing over lustre-MDT0000
[ 2572.613132] Lustre: server umount lustre-MDT0000 complete
[ 2585.521196] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2586.351626] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5266 to 0x280000401:5313
[ 2586.351628] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5298 to 0x240000401:5345
[ 2587.982087] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2588.369928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2592.904092] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 09:31:36 (1713533496)
[ 2603.839570] Lustre: Failing over lustre-OST0000
[ 2603.849913] Lustre: server umount lustre-OST0000 complete
[ 2604.688845] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 2604.693237] LustreError: Skipped 4 previous similar messages
[ 2604.695941] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2604.705055] LustreError: Skipped 4 previous similar messages
[ 2616.956223] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2619.336254] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2619.771789] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2625.354741] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 09:32:08 (1713533528)
[ 2625.866770] Lustre: Failing over lustre-MDT0000
[ 2625.958800] Lustre: server umount lustre-MDT0000 complete
[ 2628.665452] Lustre: *** cfs_fail_loc=605, val=0***
[ 2628.668790] LustreError: 5523:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffa0553a60 failed: rc = -95
[ 2628.674985] LustreError: 5523:0:(obd_config.c:797:class_setup()) setup MGS failed (-95)
[ 2628.679081] LustreError: 5523:0:(obd_mount.c:215:lustre_start_simple()) MGS setup error -95
[ 2628.684341] LustreError: 5523:0:(tgt_mount.c:135:server_deregister_mount()) MGS not registered
[ 2628.689152] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 2628.693732] LustreError: 5523:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-MDT0000
[ 2628.708995] Lustre: server umount lustre-MDT0000 complete
[ 2628.711351] LustreError: 5523:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 2630.471961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2630.474500] Lustre: Skipped 18 previous similar messages
[ 2631.276056] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2631.426581] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5315 to 0x280000401:5345
[ 2631.426593] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5347 to 0x240000401:5377
[ 2635.590381] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 09:32:18 (1713533538)
[ 2636.847861] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2637.802132] Lustre: Failing over lustre-MDT0000
[ 2637.913346] Lustre: server umount lustre-MDT0000 complete
[ 2650.895423] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2651.432864] Lustre: *** cfs_fail_loc=707, val=0***
[ 2667.459022] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnected, waiting for 1 clients in recovery for 0:53
[ 2667.577001] Lustre: lustre-MDT0000: Recovery over after 0:16, of 1 clients 1 recovered and 0 were evicted.
[ 2667.579633] Lustre: Skipped 18 previous similar messages
[ 2667.593401] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5390 to 0x240000401:5409
[ 2667.593437] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5359 to 0x280000401:5377
[ 2668.147937] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2668.523175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2673.754331] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 09:32:56 (1713533576)
[ 2696.715651] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms
[ 2707.718334] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake
[ 2720.838086] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 09:33:43 (1713533623)
[ 2744.110571] LustreError: 18489:0:(tgt_handler.c:2759:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms
[ 2755.115356] LustreError: 18489:0:(tgt_handler.c:2759:tgt_brw_write()) cfs_fail_timeout id 224 awake
[ 2759.685548] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 09:34:22 (1713533662)
[ 2782.165939] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms
[ 2787.168370] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake
[ 2798.061322] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake
[ 2811.611187] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 09:35:14 (1713533714)
[ 2897.499942] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 09:36:40 (1713533800)
[ 2920.263511] LustreError: 14472:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms
[ 2920.270313] LustreError: 14472:0:(service.c:2338:ptlrpc_server_handle_request()) Skipped 1 previous similar message
[ 2920.676322] LustreError: 14472:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake
[ 2936.663137] LustreError: 14472:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms
[ 2936.669487] LustreError: 14472:0:(service.c:2338:ptlrpc_server_handle_request()) Skipped 38 previous similar messages
[ 2937.072355] LustreError: 14472:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake
[ 2937.075527] LustreError: 14472:0:(service.c:2338:ptlrpc_server_handle_request()) Skipped 38 previous similar messages
[ 2969.049142] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms
[ 2969.053549] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) Skipped 94 previous similar messages
[ 2969.456445] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake
[ 2969.460465] LustreError: 8377:0:(service.c:2338:ptlrpc_server_handle_request()) Skipped 93 previous similar messages
[ 2973.194910] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 09:37:56 (1713533876)
[ 2997.528988] Lustre: DEBUG MARKER: phase 2
[ 3002.280253] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 09:38:25 (1713533905)
[ 3074.267543] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 09:39:37 (1713533977)
[ 3074.818810] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1
[ 3077.531962] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 09:39:40 (1713533980)
[ 3079.282407] Lustre: DEBUG MARKER: Started rundbench load pid=14013 ...
[ 3081.400352] LustreError: 24209:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3081.409054] LustreError: 24209:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages
[ 3081.751084] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3083.361791] Lustre: DEBUG MARKER: test_70b fail mds1 1 times
[ 3084.077919] Lustre: Failing over lustre-MDT0000
[ 3084.107938] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping)
[ 3084.206680] Lustre: server umount lustre-MDT0000 complete
[ 3096.645166] LustreError: 166-1: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3096.651279] LustreError: Skipped 16 previous similar messages
[ 3096.809158] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3096.816520] Lustre: Skipped 35 previous similar messages
[ 3096.876725] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3096.880421] Lustre: Skipped 7 previous similar messages
[ 3097.933028] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3101.819402] Lustre: 2638:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713533990/real 1713533990]  req@ffff8800a08ba680 x1796767410734464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1713534006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3101.833201] Lustre: 2638:0:(client.c:2340:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 3101.842344] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 3101.846120] Lustre: Skipped 36 previous similar messages
[ 3107.165004] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3107.169470] Lustre: Skipped 19 previous similar messages
[ 3107.641109] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5529 to 0x240000401:5569
[ 3107.641117] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5475 to 0x280000401:5505
[ 3108.509845] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3109.104100] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3112.811313] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3114.403647] Lustre: DEBUG MARKER: test_70b fail mds1 2 times
[ 3115.095458] Lustre: Failing over lustre-MDT0000
[ 3115.108936] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping)
[ 3115.210684] Lustre: server umount lustre-MDT0000 complete
[ 3129.026786] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3132.709441] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5568 to 0x280000401:5601
[ 3132.709509] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5632 to 0x240000401:5665
[ 3133.538797] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3134.114304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3137.869641] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3139.469691] Lustre: DEBUG MARKER: test_70b fail mds1 3 times
[ 3140.202687] Lustre: Failing over lustre-MDT0000
[ 3140.318898] Lustre: server umount lustre-MDT0000 complete
[ 3152.897033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 3152.900152] Lustre: Skipped 1 previous similar message
[ 3154.102122] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3157.664783] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5730 to 0x240000401:5761
[ 3157.664799] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5667 to 0x280000401:5697
[ 3158.357144] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3158.868335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3162.664879] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3164.239599] Lustre: DEBUG MARKER: test_70b fail mds1 4 times
[ 3164.931729] Lustre: Failing over lustre-MDT0000
[ 3165.057718] Lustre: server umount lustre-MDT0000 complete
[ 3178.766864] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3182.789395] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5834 to 0x240000401:5857
[ 3182.790085] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5769 to 0x280000401:5793
[ 3183.624183] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3184.190743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3187.930944] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3189.535019] Lustre: DEBUG MARKER: test_70b fail mds1 5 times
[ 3190.250805] Lustre: Failing over lustre-MDT0000
[ 3190.375504] Lustre: server umount lustre-MDT0000 complete
[ 3204.239382] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3207.766533] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5855 to 0x280000401:5889
[ 3207.766611] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:5920 to 0x240000401:5953
[ 3208.601083] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3209.178386] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3212.982024] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3214.603535] Lustre: DEBUG MARKER: test_70b fail mds1 6 times
[ 3215.319681] Lustre: Failing over lustre-MDT0000
[ 3215.337381] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping)
[ 3215.438254] Lustre: server umount lustre-MDT0000 complete
[ 3229.184157] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3232.859690] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:6017 to 0x240000401:6049
[ 3232.859742] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:5953 to 0x280000401:5985
[ 3233.726095] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3234.325966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3238.192854] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3239.804357] Lustre: DEBUG MARKER: test_70b fail mds1 7 times
[ 3240.501699] Lustre: Failing over lustre-MDT0000
[ 3240.617852] Lustre: server umount lustre-MDT0000 complete
[ 3253.104615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 3253.167019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3253.169557] Lustre: Skipped 7 previous similar messages
[ 3254.250088] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3257.870876] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:6050 to 0x280000401:6081
[ 3257.870880] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:6113 to 0x240000401:6145
[ 3258.777018] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3259.375308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3263.173439] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3264.787856] Lustre: DEBUG MARKER: test_70b fail mds1 8 times
[ 3265.490706] Lustre: Failing over lustre-MDT0000
[ 3265.615488] Lustre: server umount lustre-MDT0000 complete
[ 3278.192918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 3278.195996] Lustre: Skipped 1 previous similar message
[ 3279.373329] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3282.860532] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3282.865495] Lustre: Skipped 7 previous similar messages
[ 3282.887042] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:6210 to 0x240000401:6241
[ 3282.887054] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:6145 to 0x280000401:6177
[ 3283.708953] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3284.275192] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3288.064731] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3289.695096] Lustre: DEBUG MARKER: test_70b fail mds1 9 times
[ 3290.389749] Lustre: Failing over lustre-MDT0000
[ 3290.519531] Lustre: server umount lustre-MDT0000 complete
[ 3304.284195] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3307.969485] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:6307 to 0x240000401:6337
[ 3307.969653] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:6243 to 0x280000401:6273
[ 3308.829220] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3309.420970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3313.168855] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3314.744728] Lustre: DEBUG MARKER: test_70b fail mds1 10 times
[ 3315.453729] Lustre: Failing over lustre-MDT0000
[ 3315.580163] Lustre: server umount lustre-MDT0000 complete
[ 3329.272864] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3332.984097] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:6336 to 0x280000401:6369
[ 3332.984105] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:6401 to 0x240000401:6433
[ 3333.791031] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3334.378269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3338.236042] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3339.819429] Lustre: DEBUG MARKER: test_70b fail mds1 11 times
[ 3340.526903] Lustre: Failing over lustre-MDT0000
[ 3340.671133] Lustre: server umount lustre-MDT0000 complete
[ 3353.185019] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3353.193141] LustreError: Skipped 5 previous similar messages
[ 3354.529564] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3358.046208] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:6497 to 0x240000401:6529
[ 3358.046305] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:6433 to 0x280000401:6465
[ 3358.898892] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3359.482658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3363.385296] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3365.010062] Lustre: DEBUG MARKER: test_70b fail mds1 12 times
[ 3365.720975] Lustre: Failing over lustre-MDT0000
[ 3365.861522] Lustre: server umount lustre-MDT0000 complete
[ 3379.500603] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3383.033463] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:6528 to 0x280000401:6561
[ 3383.033493] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:6592 to 0x240000401:6625
[ 3383.822440] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3384.382008] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3443.848852] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 09:45:46 (1713534346)
[ 3565.406675] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3575.968228] Lustre: DEBUG MARKER: test_70c fail mds1 1 times
[ 3576.654302] Lustre: Failing over lustre-MDT0000
[ 3576.827787] Lustre: server umount lustre-MDT0000 complete
[ 3590.534228] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3596.488040] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:8885 to 0x240000401:8961
[ 3596.488084] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:8821 to 0x280000401:8865
[ 3597.243664] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3597.769430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3720.170376] LustreError: 10368:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3720.174705] LustreError: 10368:0:(osd_handler.c:698:osd_ro()) Skipped 12 previous similar messages
[ 3720.515999] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3731.101176] Lustre: DEBUG MARKER: test_70c fail mds1 2 times
[ 3731.797229] Lustre: Failing over lustre-MDT0000
[ 3731.989432] Lustre: server umount lustre-MDT0000 complete
[ 3744.459218] LustreError: 166-1: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3744.465425] LustreError: Skipped 12 previous similar messages
[ 3744.617214] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3744.630416] Lustre: Skipped 25 previous similar messages
[ 3744.738875] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3744.740871] Lustre: Skipped 12 previous similar messages
[ 3745.799074] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3747.982398] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3747.989942] Lustre: Skipped 12 previous similar messages
[ 3749.666672] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 3749.672406] Lustre: Skipped 25 previous similar messages
[ 3750.758347] Lustre: 2639:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713534639/real 1713534639]  req@ffff8800a2549f80 x1796767412456384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1713534655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 3750.772512] Lustre: 2639:0:(client.c:2340:ptlrpc_expire_one_request()) Skipped 59 previous similar messages
[ 3751.790354] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:10400 to 0x280000401:10433
[ 3751.790526] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:10496 to 0x240000401:10529
[ 3752.626380] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3753.216971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3817.904005] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 09:52:00 (1713534720)
[ 3818.434242] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs
[ 3821.232829] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 09:52:04 (1713534724)
[ 3821.804397] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs
[ 3824.644586] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 09:52:07 (1713534727)
[ 3829.351342] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3830.934906] Lustre: DEBUG MARKER: test_70f failing OST 1 times
[ 3831.630638] Lustre: Failing over lustre-OST0000
[ 3831.646894] Lustre: server umount lustre-OST0000 complete
[ 3831.936737] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3831.941774] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3843.142099] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3843.150786] LustreError: Skipped 4 previous similar messages
[ 3844.053998] mount.lustre (21211) used greatest stack depth: 9856 bytes left
[ 3845.803458] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3848.683909] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3849.216853] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3856.975431] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3858.583422] Lustre: DEBUG MARKER: test_70f failing OST 2 times
[ 3859.217524] Lustre: Failing over lustre-OST0000
[ 3859.232159] Lustre: server umount lustre-OST0000 complete
[ 3860.257000] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3860.262310] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3871.675868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3871.680421] Lustre: Skipped 8 previous similar messages
[ 3873.419371] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3876.262898] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3876.836109] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3884.536333] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3886.157920] Lustre: DEBUG MARKER: test_70f failing OST 3 times
[ 3886.859341] Lustre: Failing over lustre-OST0000
[ 3886.875612] Lustre: server umount lustre-OST0000 complete
[ 3888.577066] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3896.704976] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3896.712590] LustreError: Skipped 8 previous similar messages
[ 3899.274887] mount.lustre (1085) used greatest stack depth: 9840 bytes left
[ 3900.888581] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 3900.893075] Lustre: Skipped 8 previous similar messages
[ 3900.960298] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3903.628016] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3904.062804] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3911.846937] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3913.427134] Lustre: DEBUG MARKER: test_70f failing OST 4 times
[ 3914.113343] Lustre: Failing over lustre-OST0000
[ 3914.130950] Lustre: server umount lustre-OST0000 complete
[ 3928.273956] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3931.024498] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3931.525966] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3939.215541] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3940.792187] Lustre: DEBUG MARKER: test_70f failing OST 5 times
[ 3941.492829] Lustre: Failing over lustre-OST0000
[ 3941.507650] Lustre: server umount lustre-OST0000 complete
[ 3955.685035] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3958.415785] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3958.859643] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3967.171624] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 09:54:30 (1713534870)
[ 3967.723895] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs
[ 3970.512879] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 09:54:33 (1713534873)
[ 3972.033349] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3973.076817] Lustre: Failing over lustre-MDT0000
[ 3973.211462] Lustre: server umount lustre-MDT0000 complete
[ 3986.941210] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3987.386090] Lustre: *** cfs_fail_loc=302, val=2147483648***
[ 4003.410130] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnected, waiting for 1 clients in recovery for 0:53
[ 4003.491121] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:10837 to 0x280000401:10881
[ 4003.491135] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:10935 to 0x240000401:10977
[ 4004.365004] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4004.940297] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4010.871357] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 09:55:13 (1713534913)
[ 4012.296888] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4013.316984] Lustre: Failing over lustre-MDT0000
[ 4013.426203] Lustre: server umount lustre-MDT0000 complete
[ 4027.212822] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4027.455417] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4027.458379] LustreError: 22457:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a1abd500 x1796767427266112/t382252089347(382252089347) o101->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:532/0 lens 592/608 e 0 to 0 dl 1713534942 ref 1 fl Interpret:/204/0 rc 301/0 job:'multiop.0' uid:0 gid:0
[ 4043.468784] Lustre: lustre-MDT0000: Client 8546c19e-2c21-4e2a-a55d-ca60123a6ea0 (at 192.168.201.19@tcp) reconnected, waiting for 1 clients in recovery for 0:53
[ 4043.477376] Lustre: 23107:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88012bf2ce00 x1796767427266112/t382252089347(382252089347) o101->8546c19e-2c21-4e2a-a55d-ca60123a6ea0@192.168.201.19@tcp:548/0 lens 592/3488 e 0 to 0 dl 1713534958 ref 1 fl Interpret:/206/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 4043.530641] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:10837 to 0x280000401:10913
[ 4043.530649] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:10979 to 0x240000401:11009
[ 4044.333519] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4044.885742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4050.598662] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 09:55:53 (1713534953)
[ 4051.442638] Lustre: Failing over lustre-OST0000
[ 4051.477903] Lustre: server umount lustre-OST0000 complete
[ 4052.973779] Lustre: Failing over lustre-MDT0000
[ 4053.133133] Lustre: server umount lustre-MDT0000 complete
[ 4065.729135] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4065.735882] LustreError: Skipped 12 previous similar messages
[ 4065.787095] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:10837 to 0x280000401:10945
[ 4066.857213] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4069.442336] Lustre: lustre-OST0000: Denying connection for new client 446e5834-b3ad-4b5f-a3c7-0aa9358399d6 (at 192.168.201.19@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59
[ 4069.451314] Lustre: Skipped 10 previous similar messages
[ 4071.007908] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:10979 to 0x240000401:11041
[ 4071.088206] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4078.624896] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 09:56:21 (1713534981)
[ 4079.178652] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs
[ 4082.025501] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 09:56:24 (1713534984)
[ 4082.590849] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs
[ 4085.407297] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 09:56:28 (1713534988)
[ 4085.985081] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs
[ 4088.811608] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 09:56:31 (1713534991)
[ 4089.371029] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs
[ 4092.159405] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 09:56:35 (1713534995)
[ 4092.717355] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs
[ 4095.534979] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 09:56:38 (1713534998)
[ 4096.101235] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs
[ 4098.931519] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 09:56:41 (1713535001)
[ 4099.501777] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs
[ 4102.303355] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 09:56:45 (1713535005)
[ 4102.867320] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs
[ 4105.751738] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 09:56:48 (1713535008)
[ 4106.306175] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs
[ 4109.145415] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 09:56:52 (1713535012)
[ 4109.711593] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs
[ 4112.548177] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 09:56:55 (1713535015)
[ 4113.090766] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs
[ 4115.919851] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 09:56:58 (1713535018)
[ 4116.485228] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs
[ 4119.296007] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 09:57:02 (1713535022)
[ 4119.867925] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs
[ 4122.749177] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 09:57:05 (1713535025)
[ 4123.322073] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs
[ 4126.226172] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 09:57:09 (1713535029)
[ 4126.785312] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs
[ 4129.648909] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 09:57:12 (1713535032)
[ 4130.213144] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs
[ 4133.061863] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 09:57:16 (1713535036)
[ 4133.751838] Lustre: 3785:0:(genops.c:1659:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 446e5834-b3ad-4b5f-a3c7-0aa9358399d6 at adminstrative request
[ 4140.920825] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 09:57:23 (1713535043)
[ 4143.374696] Lustre: Failing over lustre-MDT0000
[ 4143.538082] Lustre: server umount lustre-MDT0000 complete
[ 4157.411813] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4158.149709] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:10997 to 0x280000401:11041
[ 4158.149749] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:11093 to 0x240000401:11137
[ 4160.257151] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4160.778963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4166.616778] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 09:57:49 (1713535069)
[ 4171.686788] Lustre: Failing over lustre-OST0000
[ 4171.752135] Lustre: server umount lustre-OST0000 complete
[ 4176.241055] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 4186.019066] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4188.903011] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4189.492084] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4195.642022] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 09:58:18 (1713535098)
[ 4197.061440] Lustre: Failing over lustre-MDT0000
[ 4197.202950] Lustre: server umount lustre-MDT0000 complete
[ 4199.723950] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:11238 to 0x240000401:11265
[ 4199.723955] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:10997 to 0x280000401:11073
[ 4200.777455] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4206.194489] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 09:58:29 (1713535109)
[ 4207.891758] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 4208.910655] Lustre: Failing over lustre-OST0000
[ 4208.926325] Lustre: server umount lustre-OST0000 complete
[ 4209.680656] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 4209.686219] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4209.695510] LustreError: Skipped 6 previous similar messages
[ 4222.959511] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4225.826851] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4226.365859] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4232.364937] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 09:58:55 (1713535135)
[ 4234.078087] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 4236.054782] Lustre: Failing over lustre-OST0000
[ 4236.070019] Lustre: server umount lustre-OST0000 complete
[ 4250.173526] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.19@tcp inode [0x20002da91:0x5:0x0] object 0x240000401:11267 extent [0-1048575]: client csum 8c1acec9, server csum 6069d30f
[ 4250.222942] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4253.109955] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4253.526059] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4257.570054] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 09:59:20 (1713535160)
[ 4258.599217] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 4259.571211] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4261.198903] Lustre: Failing over lustre-MDT0000
[ 4261.311765] Lustre: server umount lustre-MDT0000 complete
[ 4262.353010] LustreError: 23479:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713535166 with bad export cookie 13601000980027240991
[ 4272.353261] Lustre: Failing over lustre-OST0000
[ 4272.363303] Lustre: server umount lustre-OST0000 complete
[ 4287.488636] LustreError: 2635:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8801317c5c00 x1796767412975296/t0(0) o250->MGC192.168.201.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0
[ 4287.897114] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:10997 to 0x280000401:11105
[ 4288.328645] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4301.483576] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4307.458529] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 10:00:10 (1713535210)
[ 4316.404565] Lustre: Failing over lustre-OST0000
[ 4316.415662] Lustre: server umount lustre-OST0000 complete
[ 4316.960635] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 4317.482844] Lustre: Failing over lustre-MDT0000
[ 4317.573188] Lustre: server umount lustre-MDT0000 complete
[ 4330.558655] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4331.216865] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:10997 to 0x280000401:11137
[ 4331.260687] LustreError: 31381:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.19@tcp arrived at 1713535235 with bad export cookie 13601000980027245380
[ 4331.264200] LustreError: 31381:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 4333.636778] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4334.264187] Lustre: lustre-OST0000: Denying connection for new client b660fa11-53ec-41c7-9935-cbc03afd7c48 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07
[ 4354.293679] Lustre: lustre-OST0000: Denying connection for new client b660fa11-53ec-41c7-9935-cbc03afd7c48 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47
[ 4354.298231] Lustre: Skipped 3 previous similar messages
[ 4389.349664] Lustre: lustre-OST0000: Denying connection for new client b660fa11-53ec-41c7-9935-cbc03afd7c48 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:12
[ 4389.354118] Lustre: Skipped 6 previous similar messages
[ 4401.592367] Lustre: lustre-OST0000: recovery is timed out, evict stale exports
[ 4401.594322] Lustre: 24202:0:(genops.c:1516:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client d4ec9253-b789-4d95-b212-125c9d0764b8@<unknown>
[ 4401.597669] Lustre: lustre-OST0000: disconnecting 1 stale clients
[ 4401.609048] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.119@tcp (at 0@lo)
[ 4401.611421] Lustre: Skipped 22 previous similar messages
[ 4401.613668] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:11316 to 0x240000401:11337
[ 4405.395379] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec
[ 4414.019136] Lustre: DEBUG MARKER: free_before: 7517184 free_after: 7517184
[ 4416.768064] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 10:01:59 (1713535319)
[ 4417.783537] Lustre: Failing over lustre-OST0001
[ 4417.792602] Lustre: server umount lustre-OST0001 complete
[ 4419.808668] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 4419.810688] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4419.813592] Lustre: Skipped 20 previous similar messages
[ 4429.578376] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[ 4429.580340] Lustre: Skipped 16 previous similar messages
[ 4430.889972] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4430.893899] Lustre: Skipped 16 previous similar messages
[ 4430.921411] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4435.807890] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 10:02:18 (1713535338)
[ 4436.846752] Lustre: Failing over lustre-OST0000
[ 4436.857416] Lustre: server umount lustre-OST0000 complete
[ 4449.614540] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4450.015109] LustreError: 29369:0:(ldlm_lib.c:2829:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms
[ 4456.458378] Lustre: *** cfs_fail_loc=715, val=40***
[ 4466.013400] Lustre: 2635:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713535354/real 1713535354]  req@ffff8800a1abf800 x1796767413003264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1713535370 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0
[ 4466.025758] Lustre: 2635:0:(client.c:2340:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
[ 4466.029868] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 2 clients in recovery for 0:52
[ 4472.048345] Lustre: *** cfs_fail_loc=715, val=40***
[ 4472.049902] Lustre: Skipped 1 previous similar message
[ 4482.033663] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 2 clients in recovery for 0:36
[ 4482.036815] Lustre: Skipped 1 previous similar message
[ 4488.048341] Lustre: *** cfs_fail_loc=715, val=40***
[ 4488.049655] Lustre: Skipped 1 previous similar message
[ 4490.025352] LustreError: 29369:0:(ldlm_lib.c:2829:target_recovery_thread()) cfs_fail_timeout id 715 awake
[ 4490.546247] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4490.906439] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4495.026968] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 10:03:18 (1713535398)
[ 4496.028589] Lustre: Failing over lustre-MDT0000
[ 4496.133089] Lustre: server umount lustre-MDT0000 complete
[ 4507.616948] LustreError: 166-1: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4507.619526] LustreError: Skipped 7 previous similar messages
[ 4507.724161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4507.725820] Lustre: Skipped 18 previous similar messages
[ 4508.359846] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4509.999744] LustreError: 31926:0:(ldlm_lib.c:2829:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms
[ 4516.016332] Lustre: *** cfs_fail_loc=715, val=80***
[ 4516.017429] Lustre: Skipped 1 previous similar message
[ 4525.558678] Lustre: lustre-MDT0000: Client b660fa11-53ec-41c7-9935-cbc03afd7c48 (at 192.168.201.19@tcp) reconnected, waiting for 1 clients in recovery for 0:54
[ 4525.561296] Lustre: Skipped 1 previous similar message
[ 4531.568409] Lustre: *** cfs_fail_loc=715, val=80***
[ 4541.564239] Lustre: lustre-MDT0000: Client b660fa11-53ec-41c7-9935-cbc03afd7c48 (at 192.168.201.19@tcp) reconnected, waiting for 1 clients in recovery for 0:38
[ 4547.584400] Lustre: *** cfs_fail_loc=715, val=80***
[ 4557.573519] Lustre: lustre-MDT0000: Client b660fa11-53ec-41c7-9935-cbc03afd7c48 (at 192.168.201.19@tcp) reconnected, waiting for 1 clients in recovery for 0:22
[ 4563.600484] Lustre: *** cfs_fail_loc=715, val=80***
[ 4589.584859] Lustre: lustre-MDT0000: Recovery already passed deadline 0:09. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery.
[ 4590.001486] LustreError: 31926:0:(ldlm_lib.c:2829:target_recovery_thread()) cfs_fail_timeout id 715 awake
[ 4590.019543] Lustre: 31926:0:(ldlm_lib.c:2874:target_recovery_thread()) too long recovery - read logs
[ 4590.025999] LustreError: dumping log to /tmp/lustre-log.1713535494.31926
[ 4590.096395] Lustre: lustre-MDT0000: Recovery over after 1:20, of 1 clients 1 recovered and 0 were evicted.
[ 4590.098276] Lustre: Skipped 15 previous similar messages
[ 4590.120388] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:11351 to 0x240000401:11369
[ 4590.120462] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:11150 to 0x280000401:11169
[ 4590.816251] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4591.413557] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4597.438042] Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 10:05:00 (1713535500)
[ 4598.012486] Lustre: DEBUG MARKER: SKIP: replay-single test_100a needs >= 2 MDTs
[ 4600.951802] Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 10:05:03 (1713535503)
[ 4601.516076] Lustre: DEBUG MARKER: SKIP: replay-single test_100b needs >= 2 MDTs
[ 4604.410083] Lustre: DEBUG MARKER: == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 10:05:07 (1713535507)
[ 4604.960904] Lustre: DEBUG MARKER: SKIP: replay-single test_100c needs >= 2 MDTs
[ 4607.900382] Lustre: DEBUG MARKER: == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 10:05:10 (1713535510)
[ 4608.465195] Lustre: DEBUG MARKER: SKIP: replay-single test_100d needs > 1 MDTs
[ 4611.226831] Lustre: DEBUG MARKER: == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 10:05:14 (1713535514)
[ 4611.786226] Lustre: DEBUG MARKER: SKIP: replay-single test_100e needs >= 2 MDTs
[ 4614.763648] Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 10:05:17 (1713535517)
[ 4615.922781] LustreError: 3828:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4615.927993] LustreError: 3828:0:(osd_handler.c:698:osd_ro()) Skipped 11 previous similar messages
[ 4616.251939] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4624.421017] Lustre: Failing over lustre-MDT0000
[ 4624.612105] Lustre: server umount lustre-MDT0000 complete
[ 4627.666583] Lustre: lustre-MDT0000: Aborting client recovery
[ 4627.670250] LustreError: 5285:0:(ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 4627.676350] Lustre: 5405:0:(ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 4627.680573] Lustre: 5405:0:(ldlm_lib.c:2310:target_recovery_overseer()) Skipped 2 previous similar messages
[ 4627.684517] Lustre: 5405:0:(genops.c:1516:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client b660fa11-53ec-41c7-9935-cbc03afd7c48@<unknown>
[ 4627.691088] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 4627.708684] Lustre: lustre-MDT0000-osd: cancel update llog [0x20001a210:0x1:0x0]
[ 4627.757917] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:11351 to 0x240000401:11913
[ 4627.757919] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:11171 to 0x280000401:11713
[ 4628.576806] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4655.339678] Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 10:05:58 (1713535558)
[ 4655.700333] Lustre: *** cfs_fail_loc=159, val=0***
[ 4671.701158] Lustre: lustre-MDT0000: Client b660fa11-53ec-41c7-9935-cbc03afd7c48 (at 192.168.201.19@tcp) reconnecting
[ 4671.704396] Lustre: Skipped 2 previous similar messages
[ 4675.917515] Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 10:06:19 (1713535579)
[ 4676.410549] Lustre: *** cfs_fail_loc=15a, val=0***
[ 4676.413720] Lustre: Skipped 1 previous similar message
[ 4692.412077] Lustre: 6374:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88009ff12300 x1796767428848256/t416611831800(0) o36->b660fa11-53ec-41c7-9935-cbc03afd7c48@192.168.201.19@tcp:442/0 lens 488/3152 e 0 to 0 dl 1713535607 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0
[ 4692.417808] Lustre: 6374:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 6 previous similar messages
[ 4695.924714] Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 10:06:38 (1713535598)
[ 4697.132104] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4697.453643] Lustre: *** cfs_fail_loc=15a, val=0***
[ 4697.456512] Lustre: Skipped 5 previous similar messages
[ 4699.310798] Lustre: Failing over lustre-MDT0000
[ 4699.419323] Lustre: server umount lustre-MDT0000 complete
[ 4712.492980] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12424 to 0x240000401:12457
[ 4712.493388] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12225 to 0x280000401:12257
[ 4712.747369] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4715.029701] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4715.609625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4721.241691] Lustre: DEBUG MARKER: == replay-single test 102d: check replay
[ 4721.686505] Lustre: *** cfs_fail_loc=15a, val=0***
[ 4721.688585] Lustre: Skipped 3 previous similar messages
[ 4723.901656] Lustre: Failing over lustre-MDT0000
[ 4724.031912] Lustre: server umount lustre-MDT0000 complete
[ 4736.611430] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4737.508601] Lustre: 13182:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8800942c1c00 x1796767428873024/t420906795056(0) o36->b660fa11-53ec-41c7-9935-cbc03afd7c48@192.168.201.19@tcp:487/0 lens 488/3152 e 0 to 0 dl 1713535652 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0
[ 4737.517522] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12262 to 0x280000401:12289
[ 4737.517523] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12461 to 0x240000401:12489
[ 4739.179509] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4739.745423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4745.654549] Lustre: DEBUG MARKER: == replay-single test 103: Check otr_next_id overflow ==== 10:07:28 (1713535648)
[ 4747.096244] Lustre: Failing over lustre-MDT0000
[ 4747.244375] Lustre: server umount lustre-MDT0000 complete
[ 4761.066130] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4762.580249] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12305 to 0x280000401:12321
[ 4762.580261] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12505 to 0x240000401:12521
[ 4763.935919] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4764.457715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4770.364131] Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 10:07:53 (1713535673)
[ 4770.913247] Lustre: DEBUG MARKER: SKIP: replay-single test_110a needs >= 2 MDTs
[ 4773.877550] Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 10:07:56 (1713535676)
[ 4774.432817] Lustre: DEBUG MARKER: SKIP: replay-single test_110b needs >= 2 MDTs
[ 4777.330755] Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 10:08:00 (1713535680)
[ 4777.903974] Lustre: DEBUG MARKER: SKIP: replay-single test_110c needs >= 2 MDTs
[ 4780.735382] Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 10:08:03 (1713535683)
[ 4781.333676] Lustre: DEBUG MARKER: SKIP: replay-single test_110d needs >= 2 MDTs
[ 4784.275461] Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 10:08:07 (1713535687)
[ 4784.801519] Lustre: DEBUG MARKER: SKIP: replay-single test_110e needs >= 2 MDTs
[ 4786.269575] Lustre: DEBUG MARKER: SKIP: replay-single test_110f skipping excluded test 110f
[ 4788.281069] Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 10:08:11 (1713535691)
[ 4788.870641] Lustre: DEBUG MARKER: SKIP: replay-single test_110g needs >= 2 MDTs
[ 4791.758114] Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 10:08:14 (1713535694)
[ 4792.291152] Lustre: DEBUG MARKER: SKIP: replay-single test_111a needs >= 2 MDTs
[ 4794.875741] Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 10:08:17 (1713535697)
[ 4795.425830] Lustre: DEBUG MARKER: SKIP: replay-single test_111b needs >= 2 MDTs
[ 4798.156471] Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 10:08:21 (1713535701)
[ 4798.667518] Lustre: DEBUG MARKER: SKIP: replay-single test_111c needs >= 2 MDTs
[ 4801.493911] Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 10:08:24 (1713535704)
[ 4801.916734] Lustre: DEBUG MARKER: SKIP: replay-single test_111d needs >= 2 MDTs
[ 4804.035701] Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 10:08:27 (1713535707)
[ 4804.437205] Lustre: DEBUG MARKER: SKIP: replay-single test_111e needs >= 2 MDTs
[ 4806.630262] Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 10:08:29 (1713535709)
[ 4807.028558] Lustre: DEBUG MARKER: SKIP: replay-single test_111f needs >= 2 MDTs
[ 4809.077499] Lustre: DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 10:08:32 (1713535712)
[ 4809.446161] Lustre: DEBUG MARKER: SKIP: replay-single test_111g needs >= 2 MDTs
[ 4811.488376] Lustre: DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 10:08:34 (1713535714)
[ 4811.874697] Lustre: DEBUG MARKER: SKIP: replay-single test_112a needs >= 4 MDTs
[ 4814.009792] Lustre: DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 10:08:37 (1713535717)
[ 4814.397972] Lustre: DEBUG MARKER: SKIP: replay-single test_112b needs >= 4 MDTs
[ 4816.447574] Lustre: DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 10:08:39 (1713535719)
[ 4816.802838] Lustre: DEBUG MARKER: SKIP: replay-single test_112c needs >= 4 MDTs
[ 4819.449065] Lustre: DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 10:08:42 (1713535722)
[ 4820.021325] Lustre: DEBUG MARKER: SKIP: replay-single test_112d needs >= 4 MDTs
[ 4822.987368] Lustre: DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 10:08:45 (1713535725)
[ 4823.417225] Lustre: DEBUG MARKER: SKIP: replay-single test_112e needs >= 4 MDTs
[ 4826.372860] Lustre: DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 10:08:49 (1713535729)
[ 4826.937007] Lustre: DEBUG MARKER: SKIP: replay-single test_112f needs >= 4 MDTs
[ 4829.888983] Lustre: DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 10:08:52 (1713535732)
[ 4830.491015] Lustre: DEBUG MARKER: SKIP: replay-single test_112g needs >= 4 MDTs
[ 4833.429626] Lustre: DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 10:08:56 (1713535736)
[ 4834.005818] Lustre: DEBUG MARKER: SKIP: replay-single test_112h needs >= 4 MDTs
[ 4836.571766] Lustre: DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 10:08:59 (1713535739)
[ 4837.126607] Lustre: DEBUG MARKER: SKIP: replay-single test_112i needs >= 4 MDTs
[ 4839.824006] Lustre: DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 10:09:02 (1713535742)
[ 4840.358365] Lustre: DEBUG MARKER: SKIP: replay-single test_112j needs >= 4 MDTs
[ 4842.710358] Lustre: DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 10:09:05 (1713535745)
[ 4843.047685] Lustre: DEBUG MARKER: SKIP: replay-single test_112k needs >= 4 MDTs
[ 4845.349027] Lustre: DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 10:09:08 (1713535748)
[ 4845.957062] Lustre: DEBUG MARKER: SKIP: replay-single test_112l needs >= 4 MDTs
[ 4847.957362] Lustre: DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 10:09:11 (1713535751)
[ 4848.419180] Lustre: DEBUG MARKER: SKIP: replay-single test_112m needs >= 4 MDTs
[ 4851.203244] Lustre: DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 10:09:14 (1713535754)
[ 4851.695820] Lustre: DEBUG MARKER: SKIP: replay-single test_112n needs >= 4 MDTs
[ 4854.105151] Lustre: DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory ========================================================== 10:09:17 (1713535757)
[ 4854.630393] Lustre: DEBUG MARKER: SKIP: replay-single test_115 needs >= 2 MDTs
[ 4857.524520] Lustre: DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery ========================================================== 10:09:20 (1713535760)
[ 4857.976783] Lustre: DEBUG MARKER: SKIP: replay-single test_116a needs >= 2 MDTs
[ 4860.412266] Lustre: DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery ========================================================== 10:09:23 (1713535763)
[ 4860.803993] Lustre: DEBUG MARKER: SKIP: replay-single test_116b needs >= 2 MDTs
[ 4862.672595] Lustre: DEBUG MARKER: == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 10:09:25 (1713535765)
[ 4863.018458] Lustre: DEBUG MARKER: SKIP: replay-single test_117 needs >= 4 MDTs
[ 4864.889064] Lustre: DEBUG MARKER: == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 10:09:28 (1713535768)
[ 4865.232629] Lustre: DEBUG MARKER: SKIP: replay-single test_118 needs >= 2 MDTs
[ 4867.067580] Lustre: DEBUG MARKER: == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 10:09:30 (1713535770)
[ 4867.392175] Lustre: DEBUG MARKER: SKIP: replay-single test_119 needs >= 2 MDTs
[ 4869.258796] Lustre: DEBUG MARKER: == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 10:09:32 (1713535772)
[ 4869.628133] Lustre: DEBUG MARKER: SKIP: replay-single test_120 needs >= 2 MDTs
[ 4871.509443] Lustre: DEBUG MARKER: == replay-single test 121: lock replay timed out and race ========================================================== 10:09:34 (1713535774)
[ 4872.219781] Lustre: Failing over lustre-MDT0000
[ 4872.320519] Lustre: server umount lustre-MDT0000 complete
[ 4874.732537] Lustre: *** cfs_fail_loc=721, val=0***
[ 4874.733611] Lustre: Skipped 6 previous similar messages
[ 4875.597395] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4876.234677] Lustre: *** cfs_fail_loc=721, val=0***
[ 4876.236401] Lustre: Skipped 29 previous similar messages
[ 4878.842436] Lustre: *** cfs_fail_loc=721, val=1***
[ 4878.844832] Lustre: Skipped 31 previous similar messages
[ 4884.864809] Lustre: *** cfs_fail_loc=721, val=1***
[ 4884.867188] Lustre: Skipped 23 previous similar messages
[ 4892.720104] Lustre: lustre-MDT0000: Client b660fa11-53ec-41c7-9935-cbc03afd7c48 (at 192.168.201.19@tcp) reconnected, waiting for 1 clients in recovery for 0:53
[ 4892.726669] Lustre: Skipped 1 previous similar message
[ 4892.733742] Lustre: *** cfs_fail_loc=721, val=1***
[ 4892.736685] Lustre: Skipped 6 previous similar messages
[ 4892.740583] Lustre: 3567:0:(service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (11/5s); client may timeout  req@ffff880087937800 x1796767428904576/t0(0) o101->b660fa11-53ec-41c7-9935-cbc03afd7c48@192.168.201.19@tcp:0/0 lens 328/0 e 0 to 0 dl 1713535792 ref 1 fl Complete:n/240/ffffffff rc -116/-1 job:'ldlm_lock_repla.0' uid:0 gid:0
[ 4893.753118] Lustre: *** cfs_fail_loc=721, val=0***
[ 4893.755549] Lustre: Skipped 18 previous similar messages
[ 4893.777673] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12505 to 0x240000401:12553
[ 4893.777686] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12323 to 0x280000401:12353
[ 4898.862615] Lustre: DEBUG MARKER: == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 10:10:01 (1713535801)
[ 4900.289628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4900.969650] Lustre: Failing over lustre-MDT0000
[ 4901.097808] Lustre: server umount lustre-MDT0000 complete
[ 4915.082665] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4917.235202] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12323 to 0x280000401:12385
[ 4917.235205] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12505 to 0x240000401:12585
[ 4918.118412] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4918.710699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4924.842777] Lustre: DEBUG MARKER: == replay-single test 130b: DoM file create (inherited) replay ========================================================== 10:10:27 (1713535827)
[ 4926.337924] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4927.101944] Lustre: Failing over lustre-MDT0000
[ 4927.262280] Lustre: server umount lustre-MDT0000 complete
[ 4941.237209] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4942.889730] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12505 to 0x240000401:12617
[ 4942.889774] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12323 to 0x280000401:12417
[ 4944.068199] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4944.693314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4950.817375] Lustre: DEBUG MARKER: == replay-single test 131a: DoM file write lock replay === 10:10:53 (1713535853)
[ 4952.323402] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4953.108887] Lustre: Failing over lustre-MDT0000
[ 4953.285007] Lustre: server umount lustre-MDT0000 complete
[ 4967.389849] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4967.924500] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12323 to 0x280000401:12449
[ 4967.924529] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12505 to 0x240000401:12649
[ 4970.278424] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4970.849253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4975.367736] Lustre: DEBUG MARKER: SKIP: replay-single test_131b skipping excluded test 131b
[ 4977.433970] Lustre: DEBUG MARKER: == replay-single test 132a: PFL new component instantiate replay ========================================================== 10:11:20 (1713535880)
[ 4978.895022] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4979.753252] Lustre: Failing over lustre-MDT0000
[ 4979.898470] Lustre: server umount lustre-MDT0000 complete
[ 4992.926695] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12451 to 0x280000401:12481
[ 4992.926696] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12652 to 0x240000401:12681
[ 4993.115685] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4995.443203] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4995.861180] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5001.168478] Lustre: DEBUG MARKER: == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 10:11:44 (1713535904)
[ 5001.643078] Lustre: DEBUG MARKER: SKIP: replay-single test_133 needs >= 2 MDTs
[ 5003.928856] Lustre: DEBUG MARKER: == replay-single test 134: replay creation of a file created in a pool ========================================================== 10:11:47 (1713535907)
[ 5009.460298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5010.248910] Lustre: Failing over lustre-MDT0000
[ 5010.417008] Lustre: server umount lustre-MDT0000 complete
[ 5022.845750] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5022.855745] Lustre: Skipped 22 previous similar messages
[ 5023.026322] Lustre: lustre-OST0001: deleting orphan objects from 0x280000401:12484 to 0x280000401:12513
[ 5023.026324] Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:12652 to 0x240000401:12713
[ 5024.014622] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5026.950779] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5027.594290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5027.874548] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 5027.878010] Lustre: Skipped 22 previous similar messages
[ 5040.181887] Lustre: DEBUG MARKER: == replay-single test 135: Server failure in lock replay phase ========================================================== 10:12:23 (1713535943)
[ 5043.022912] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5043.853172] Lustre: Failing over lustre-OST0000
[ 5043.870019] Lustre: server umount lustre-OST0000 complete
[ 5046.028667] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_module ../libcfs/libcfs/libcfs
[ 5047.888788] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5047.894803] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5047.906658] LustreError: Skipped 36 previous similar messages
[ 5048.032898] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 5048.036935] Lustre: Skipped 14 previous similar messages
[ 5049.110821] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5049.114982] Lustre: Skipped 11 previous similar messages
[ 5049.532160] Lustre: *** cfs_fail_loc=32d, val=20***
[ 5049.923813] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5051.542133] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5052.096163] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec
[ 5052.672171] Lustre: DEBUG MARKER: replay-single test_135: @@@@@@ FAIL: Unexpected sync success