== conf-sanity test 69: replace an OST with the same index ========================================================== 04:48:18 (1713343698) start mds service on oleg237-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg237-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg237-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock oleg237-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 5461 (time 1713343741.52 total 10.00 last 546.04) total: 10000 open/close in 18.16 seconds: 550.71 ops/second - unlinked 0 (time 1713343750 ; total 0 ; last 0) total: 10000 unlinks in 9 seconds: 1111.111084 unlinks/second - open/close 5878 (time 1713343770.69 total 10.00 last 587.77) total: 10000 open/close in 17.05 seconds: 586.45 ops/second - unlinked 0 (time 1713343778 ; total 0 ; last 0) total: 10000 unlinks in 9 seconds: 1111.111084 unlinks/second - open/close 5381 (time 1713343798.82 total 10.00 last 538.07) total: 10000 open/close in 17.98 seconds: 556.22 ops/second - unlinked 0 (time 1713343807 ; total 0 ; last 0) total: 10000 unlinks in 9 seconds: 1111.111084 unlinks/second - open/close 5784 (time 1713343827.36 total 10.00 last 578.33) total: 10000 open/close in 17.48 seconds: 572.13 ops/second - unlinked 0 (time 1713343835 ; total 0 ; last 0) total: 10000 unlinks in 9 seconds: 1111.111084 unlinks/second - open/close 6982 (time 1713343854.95 total 10.00 last 698.18) total: 10000 open/close in 14.42 seconds: 693.34 ops/second - unlinked 0 (time 1713343860 ; total 0 ; last 0) total: 10000 unlinks in 8 seconds: 1250.000000 unlinks/second - open/close 6590 (time 1713343878.98 total 10.00 last 658.98) total: 10000 open/close in 15.25 seconds: 655.61 ops/second - unlinked 0 (time 1713343885 ; total 0 ; last 0) total: 10000 unlinks in 8 seconds: 1250.000000 unlinks/second - open/close 6111 (time 1713343904.13 total 10.00 last 611.08) total: 10000 open/close in 15.61 seconds: 640.57 ops/second - unlinked 0 (time 1713343910 ; total 0 ; last 0) total: 10000 unlinks in 8 seconds: 1250.000000 unlinks/second - open/close 6684 (time 1713343929.28 total 10.01 last 667.67) total: 10000 open/close in 15.47 seconds: 646.34 ops/second - unlinked 0 (time 1713343935 ; total 0 ; last 0) total: 10000 unlinks in 8 seconds: 1250.000000 unlinks/second - open/close 5995 (time 1713343954.82 total 10.00 last 599.43) total: 10000 open/close in 16.81 seconds: 594.79 ops/second - unlinked 0 (time 1713343962 ; total 0 ; last 0) total: 10000 unlinks in 9 seconds: 1111.111084 unlinks/second - open/close 6857 (time 1713343982.33 total 10.00 last 685.60) total: 10000 open/close in 15.17 seconds: 659.34 ops/second - unlinked 0 (time 1713343988 ; total 0 ; last 0) total: 10000 unlinks in 8 seconds: 1250.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg237-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 12 sec oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg237-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock oleg237-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg237-server LNET ready to unload unloading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: LNET ready to unload modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2