== conf-sanity test 35a: Reconnect to the last active server first ========================================================== 15:05:02 (1713294302) start mds service on oleg447-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg447-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre debug=ha Set up a fake failnode for the MDS Wait for RECONNECT_INTERVAL seconds (10s) conf-sanity.sh test_35a 2024-04-1615h05m41s Stopping the MDT: lustre-MDT0000 stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server Restarting the MDT: lustre-MDT0000 start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 Wait for df (31684) ... done debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout Debug log: 99 lines, 99 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg447-server unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local modules unloaded. oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: oleg447-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg447-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg447-server