== sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 05:10:35 (1713431435) Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Waiting 90s for 'u' Updated after 3s: want 'u' got 'u' User quota (limit: 200) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 0 0 204800 - 0 0 0 - lustre-MDT0000_UUID 0 - 0 - 0 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 Write 100M (buffered) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d18.sanity-quota/f18.sanity-quota] [count=100] UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2884 1284804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1904 1285784 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1604 3601236 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3128 7206732 1% /mnt/lustre Fail mds for 40 seconds Failing mds1 on oleg432-server Stopping /mnt/lustre-mds1 (opts:) on oleg432-server 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 2.60689 s, 40.2 MB/s 05:10:45 (1713431445) shut down Failover mds1 to oleg432-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 05:10:59 (1713431459) targets are mounted 05:10:59 (1713431459) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec (dd_pid=18552, time=0, timeout=600) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 102400 0 204800 - 1 0 0 - lustre-MDT0000_UUID 0 - 0 - 1 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 102400 - 114688 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 114688 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (limit: 200) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 0 0 204800 - 0 0 0 - lustre-MDT0000_UUID 0 - 0 - 0 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 Write 100M (directio) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d18.sanity-quota/f18.sanity-quota] [count=100] [oflag=direct] UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1904 1285784 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1604 3593888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3128 7199384 1% /mnt/lustre Fail mds for 40 seconds 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 2.08451 s, 50.3 MB/s Failing mds1 on oleg432-server Stopping /mnt/lustre-mds1 (opts:) on oleg432-server 05:11:16 (1713431476) shut down Failover mds1 to oleg432-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 05:11:30 (1713431490) targets are mounted 05:11:30 (1713431490) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec (dd_pid=20833, time=0, timeout=600) Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 102400 0 204800 - 1 0 0 - lustre-MDT0000_UUID 0 - 0 - 1 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 102400 - 109568 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 109568 Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete Delete files... Wait for unlink objects finished... Waiting for MDT destroys to complete