== conf-sanity test 69: replace an OST with the same index =========================================== 04:38:19 (1713256699) start mds service on oleg405-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: detected 4 online CPUs by sysfs oleg405-server: Force libcfs to create 2 CPU partitions oleg405-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg405-server Starting mds2: /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg405-server Starting ost1: /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock oleg405-server@tcp:/lustre /mnt/lustre On OST0, 49539 inodes available. Want 99872. - open/close 2397 (time 1713256740.55 total 10.05 last 238.43) - open/close 4687 (time 1713256750.55 total 20.05 last 228.99) - open/close 7089 (time 1713256760.55 total 30.06 last 240.13) - open/close 9048 (time 1713256770.56 total 40.07 last 195.69) total: 10000 open/close in 44.45 seconds: 224.98 ops/second - unlinked 0 (time 1713256774 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1673 (time 1713256828.33 total 10.00 last 167.22) - open/close 3036 (time 1713256838.33 total 20.01 last 136.26) - open/close 4537 (time 1713256848.34 total 30.01 last 150.03) - open/close 6271 (time 1713256858.34 total 40.02 last 173.33) - open/close 8289 (time 1713256868.34 total 50.02 last 201.74) total: 10000 open/close in 58.89 seconds: 169.81 ops/second - unlinked 0 (time 1713256877 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1326 (time 1713256927.04 total 10.00 last 132.56) - open/close 2690 (time 1713256937.04 total 20.01 last 136.31) - open/close 4086 (time 1713256947.05 total 30.01 last 139.59) - open/close 5485 (time 1713256957.05 total 40.01 last 139.87) - open/close 6749 (time 1713256967.05 total 50.02 last 126.35) - open/close 8175 (time 1713256977.05 total 60.02 last 142.58) - open/close 9589 (time 1713256987.06 total 70.03 last 141.27) total: 10000 open/close in 72.78 seconds: 137.40 ops/second - unlinked 0 (time 1713256989 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1664 (time 1713257038.73 total 10.00 last 166.33) - open/close 3188 (time 1713257048.73 total 20.01 last 152.36) - open/close 4635 (time 1713257058.73 total 30.01 last 144.69) - open/close 6272 (time 1713257068.73 total 40.01 last 163.67) - open/close 7833 (time 1713257078.73 total 50.01 last 156.09) - open/close 9446 (time 1713257088.74 total 60.02 last 161.21) total: 10000 open/close in 64.37 seconds: 155.36 ops/second - unlinked 0 (time 1713257093 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1672 (time 1713257144.17 total 10.01 last 167.11) - open/close 3450 (time 1713257154.18 total 20.01 last 177.77) - open/close 5192 (time 1713257164.18 total 30.01 last 174.08) - open/close 6738 (time 1713257174.19 total 40.02 last 154.52) - open/close 8347 (time 1713257184.19 total 50.02 last 160.83) - open/close 9928 (time 1713257194.20 total 60.03 last 158.03) total: 10000 open/close in 60.40 seconds: 165.55 ops/second - unlinked 0 (time 1713257194 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1590 (time 1713257238.58 total 10.00 last 158.93) - open/close 3238 (time 1713257248.58 total 20.01 last 164.78) - open/close 4923 (time 1713257258.59 total 30.01 last 168.46) - open/close 6509 (time 1713257268.59 total 40.01 last 158.54) - open/close 8083 (time 1713257278.59 total 50.01 last 157.37) - open/close 9753 (time 1713257288.59 total 60.02 last 166.94) total: 10000 open/close in 61.45 seconds: 162.73 ops/second - unlinked 0 (time 1713257290 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1676 (time 1713257337.32 total 10.01 last 167.51) - open/close 3651 (time 1713257347.33 total 20.01 last 197.42) - open/close 5849 (time 1713257357.33 total 30.01 last 219.75) - open/close 7931 (time 1713257367.33 total 40.01 last 208.16) - open/close 9882 (time 1713257377.33 total 50.02 last 195.03) total: 10000 open/close in 50.73 seconds: 197.10 ops/second - unlinked 0 (time 1713257378 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1673 (time 1713257424.31 total 10.00 last 167.30) - open/close 3347 (time 1713257434.31 total 20.00 last 167.33) - open/close 5224 (time 1713257444.31 total 30.00 last 187.70) - open/close 7021 (time 1713257454.32 total 40.01 last 179.60) - open/close 8959 (time 1713257464.32 total 50.01 last 193.77) total: 10000 open/close in 54.65 seconds: 182.97 ops/second - unlinked 0 (time 1713257468 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1920 (time 1713257509.04 total 10.00 last 191.97) - open/close 3710 (time 1713257519.04 total 20.00 last 178.96) - open/close 5697 (time 1713257529.05 total 30.01 last 198.67) - open/close 7663 (time 1713257539.05 total 40.01 last 196.53) - open/close 9605 (time 1713257549.05 total 50.01 last 194.16) total: 10000 open/close in 52.00 seconds: 192.30 ops/second - unlinked 0 (time 1713257551 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1805 (time 1713257596.28 total 10.00 last 180.46) - open/close 3593 (time 1713257606.29 total 20.01 last 178.70) - open/close 5250 (time 1713257616.29 total 30.01 last 165.61) - open/close 6774 (time 1713257626.30 total 40.02 last 152.31) - open/close 8346 (time 1713257636.30 total 50.02 last 157.16) total: 10000 open/close in 59.30 seconds: 168.63 ops/second - unlinked 0 (time 1713257645 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,large_dir,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,large_dir,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg405-server Starting ost1: /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg405-server: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg405-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg405-server: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg405-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock oleg405-server@tcp:/lustre /mnt/lustre On OST0, 10428 used inodes umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg405-server unloading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local modules unloaded. Resetting fail_loc on all nodes...pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2 done. 04:55:20 (1713257720) waiting for oleg405-server network 5 secs ... 04:55:20 (1713257720) network interface is UP