-----============= acceptance-small: sanity-lnet ============----- Fri Apr 19 08:35:00 EDT 2024 Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:) Stopping client oleg214-client.virtnet /mnt/lustre opts: Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg214-server unloading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local modules unloaded. Writer error: failed to resolve Netlink family id Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) LNET unconfigure error 22: (null) unloading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local oleg214-server: LNET unconfigure error 22: (null) modules unloaded. ip netns exec test_ns ip addr add 10.1.2.3/31 dev test1pg ip netns exec test_ns ip link set test1pg up Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.14@tcp status: up interfaces: 0: eth0 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 9000 qdisc pfifo_fast state UP group default qlen 1000 link/ether 56:4b:0b:d3:34:f2 brd ff:ff:ff:ff:ff:ff inet 192.168.202.14/24 brd 192.168.202.255 scope global dynamic eth0 valid_lft 59914sec preferred_lft 59914sec inet6 fe80::544b:bff:fed3:34f2/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: test1pl@if2: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether e2:38:cd:6b:e6:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::e038:cdff:fe6b:e637/64 scope link tentative valid_lft forever preferred_lft forever excepting tests: 253 254 Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local oleg214-server: LNET unconfigure error 22: (null) modules unloaded. error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:35:21 (1713530121) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 0 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:35:25 (1713530125) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 1 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:35:29 (1713530129) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib Expected: peer: - primary nid: 2.2.2.2@o2ib Multi-Rail: True peer ni: - nid: 2.2.2.2@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 2.2.2.2@o2ib Multi-Rail: True peer ni: - nid: 2.2.2.2@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 2 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:35:32 (1713530132) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib Expected: peer: - primary nid: 3.3.3.3@tcp Multi-Rail: True peer ni: - nid: 3.3.3.3@tcp - nid: 3.3.3.3@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 3.3.3.3@tcp Multi-Rail: True peer ni: - nid: 3.3.3.3@tcp - nid: 3.3.3.3@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 3 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:35:36 (1713530136) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp Expected: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidrange that overlaps primary nid (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp Expected: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 4 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:35:39 (1713530139) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidranage that overlaps primary nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidranage that contain , plus primary nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 5 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 6: Add peer with multiple nidranges == 08:35:43 (1713530143) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi Expected: peer: - primary nid: 6.6.6.6@tcp Multi-Rail: True peer ni: - nid: 6.6.6.6@tcp - nid: 6.6.6.0@tcp - nid: 6.6.6.2@tcp - nid: 6.6.6.4@tcp - nid: 6.6.7.0@tcp - nid: 6.6.7.2@tcp - nid: 6.6.7.4@tcp - nid: 6.6.1.0@o2ib - nid: 6.6.1.3@o2ib - nid: 6.6.1.6@o2ib - nid: 6.6.3.0@o2ib - nid: 6.6.3.3@o2ib - nid: 6.6.3.6@o2ib - nid: 6@gni - nid: 10@gni - nid: 6@kfi - nid: 10@kfi global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 6.6.6.6@tcp Multi-Rail: True peer ni: - nid: 6.6.6.6@tcp - nid: 6.6.6.0@tcp - nid: 6.6.6.2@tcp - nid: 6.6.6.4@tcp - nid: 6.6.7.0@tcp - nid: 6.6.7.2@tcp - nid: 6.6.7.4@tcp - nid: 6.6.1.0@o2ib - nid: 6.6.1.3@o2ib - nid: 6.6.1.6@o2ib - nid: 6.6.3.0@o2ib - nid: 6.6.3.3@o2ib - nid: 6.6.3.6@o2ib - nid: 6@gni - nid: 10@gni - nid: 6@kfi - nid: 10@kfi global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 6 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 7: Various peer delete tests ========= 08:35:46 (1713530146) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Delete peer with single nid (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has both tcp and o2ib nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (kfi) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (kfi) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has tcp, o2ib, gni and kfi nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that contains nid equal to primary nid /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 7 (4s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:35:52 (1713530152) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp Expected: peer: - primary nid: 8.8.8.8@tcp Multi-Rail: True peer ni: - nid: 8.8.8.8@tcp - nid: 8.8.8.10@tcp - nid: 8.8.8.11@tcp - nid: 8.8.8.12@tcp - nid: 8.8.8.14@tcp - nid: 8.8.8.15@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 8.8.8.8@tcp Multi-Rail: True peer ni: - nid: 8.8.8.8@tcp - nid: 8.8.8.10@tcp - nid: 8.8.8.11@tcp - nid: 8.8.8.12@tcp - nid: 8.8.8.14@tcp - nid: 8.8.8.15@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 8 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:35:55 (1713530155) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp Expected: peer: - primary nid: 9.9.9.9@tcp Multi-Rail: True peer ni: - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 9.9.9.9@tcp Multi-Rail: True peer ni: - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 9 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:35:58 (1713530158) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp Expected: peer: - primary nid: 10.10.10.10@tcp Multi-Rail: True peer ni: - nid: 10.10.10.10@tcp - nid: 10.10.10.12@tcp - nid: 10.10.10.13@tcp - nid: 10.10.10.15@tcp - nid: 10.10.10.16@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 10.10.10.10@tcp Multi-Rail: True peer ni: - nid: 10.10.10.10@tcp - nid: 10.10.10.12@tcp - nid: 10.10.10.13@tcp - nid: 10.10.10.15@tcp - nid: 10.10.10.16@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 10 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:36:02 (1713530162) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp Expected: peer: - primary nid: 11.11.11.11@tcp Multi-Rail: True peer ni: - nid: 11.11.11.11@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 11.11.11.11@tcp Multi-Rail: True peer ni: - nid: 11.11.11.11@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 11 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:36:05 (1713530165) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp Expected: peer: - primary nid: 12.12.12.12@o2ib Multi-Rail: True peer ni: - nid: 12.12.12.12@o2ib - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 12.12.12.12@o2ib Multi-Rail: True peer ni: - nid: 12.12.12.12@o2ib - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 12 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:36:08 (1713530168) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib Expected: peer: - primary nid: 13.13.13.13@o2ib Multi-Rail: True peer ni: - nid: 13.13.13.13@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 13.13.13.13@o2ib Multi-Rail: True peer ni: - nid: 13.13.13.13@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 13 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:36:12 (1713530172) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create single peer, single nid, using import Generating peer yaml for 1 peers with 0 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete single peer using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 14 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:36:15 (1713530175) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create multiple peers, single nid per peer, using import Generating peer yaml for 5 peers with 0 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete multiple peers, single nid per peer, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 15 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:36:18 (1713530178) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create single peer, multiple nids, using import Generating peer yaml for 1 peers with 5 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete single peer, multiple nids, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 16 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:36:21 (1713530181) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create multiple peers, multiple nids per peer, using import Generating peer yaml for 5 peers with 7 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete multiple peers, multiple nids per peer, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 17 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:36:25 (1713530185) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Import peer with 5 nids peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 2.2.2.2@tcp - nid: 4.4.4.4@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete three of the nids peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 2.2.2.2@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Check peer has expected nids remaining Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 18a (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:36:28 (1713530188) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Import two peers with 5 nids each peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 2.2.2.2@tcp - nid: 4.4.4.4@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 6.6.6.6@o2ib - nid: 7.7.7.7@tcp - nid: 8.8.8.8@tcp - nid: 9.9.9.9@tcp - nid: 10@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete three of the nids from each peer peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 2.2.2.2@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 7.7.7.7@tcp - nid: 8.8.8.8@tcp - nid: 10@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Check peers have expected nids remaining Expected: peer: - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 6.6.6.6@o2ib - nid: 7.7.7.7@tcp - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 6.6.6.6@o2ib - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 18b (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 19: Add peer with single nid (gni) === 08:36:31 (1713530191) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni Expected: peer: - primary nid: 19@gni Multi-Rail: True peer ni: - nid: 19@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 19@gni Multi-Rail: True peer ni: - nid: 19@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 19 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:36:34 (1713530194) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib Expected: peer: - primary nid: 20@gni Multi-Rail: True peer ni: - nid: 20@gni - nid: 20.20.20.20@tcp - nid: 20.20.20.20@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 20@gni Multi-Rail: True peer ni: - nid: 20@gni - nid: 20.20.20.20@tcp - nid: 20.20.20.20@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 20 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:36:37 (1713530197) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni Expected: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidrange that overlaps primary nid (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni Expected: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 21 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:36:40 (1713530200) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni Expected: peer: - primary nid: 22@gni Multi-Rail: True peer ni: - nid: 22@gni - nid: 24@gni - nid: 25@gni - nid: 27@gni - nid: 28@gni - nid: 29@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 22@gni Multi-Rail: True peer ni: - nid: 22@gni - nid: 24@gni - nid: 25@gni - nid: 27@gni - nid: 28@gni - nid: 29@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 22 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:36:44 (1713530204) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni Expected: peer: - primary nid: 23@gni Multi-Rail: True peer ni: - nid: 23@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 23@gni Multi-Rail: True peer ni: - nid: 23@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 23 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:36:47 (1713530207) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp Expected: peer: - primary nid: 24@gni Multi-Rail: True peer ni: - nid: 24@gni - nid: 11@gni - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 24@gni Multi-Rail: True peer ni: - nid: 24@gni - nid: 11@gni - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 24 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:36:50 (1713530210) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni Expected: peer: - primary nid: 25@gni Multi-Rail: True peer ni: - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 25@gni Multi-Rail: True peer ni: - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 25 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:36:53 (1713530213) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp --- show: - peer: errno: -2 descr: ! "Object not found" ... pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 26 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:36:56 (1713530216) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Invalid prim_nid - peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Invalid prim_nid - peer del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar --- del: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Delete non-existing peer /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib --- del: - peer: errno: -2 descr: ! "Object not found" ... Don't provide mandatory argument for peer del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp --- del: - peer: errno: -42 descr: ! "Unspecific failure" ... Don't provide mandatory argument for peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp --- add: - peer: errno: -42 descr: ! "Unspecific failure" ... Don't provide mandatory arguments peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add peer add: add a peer NID --prim_nid: Primary NID of the peer. --nid: one or more peer NIDs --non_mr: create this peer as not Multi-Rail capable --ip2nets: specify a range of nids per peer --lock_prim: lock primary nid Invalid secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar --- add: - peer: errno: -22 descr: ! "Unable to parse nidlist: no valid NIDs in string" ... can't parse nidrange: "foobar" Exceed max nids per peer /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp --- add: - peer: errno: -7 descr: ! "Unable to parse nidlist: specifies more NIDs than allowed" ... Invalid net type /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Invalid nid format Check invalid primary nid - '1@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Check invalid primary nid - '1@o2ib' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Check invalid primary nid - '1.1.1.1@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Check invalid nidstring - '[2-1]@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni --- add: - peer: errno: -34 descr: ! "Unable to parse nidlist: range is wrong order" ... can't parse nidrange: "[2-1]" Check invalid nidstring - '[a-f/x]@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni --- add: - peer: errno: -34 descr: ! "Unable to parse nidlist: range needs number after /" ... can't parse nidrange: "[a-f/x]" Check invalid nidstring - '256.256.256.256@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Check invalid nidstring - '1.1.1.1.[2-5/f]@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp --- add: - peer: errno: -34 descr: ! "Unable to parse nidlist: range needs number after /" ... can't parse nidrange: "1.1.1.1.[2-5/f]" Check invalid nidstring - '1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp --- add: - peer: errno: -22 descr: ! "Unable to parse nidlist: improper bracket ordering" ... can't parse nidrange: "1.]2[.3.4" Check invalid nidstring - 'foobar' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar --- add: - peer: errno: -22 descr: ! "Unable to parse nidlist: no valid NIDs in string" ... can't parse nidrange: "foobar" Add non-local gateway /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni --- add: - route: errno: -113 descr: ! "Unspecific failure" ... pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 99a (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:36:59 (1713530219) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import peer: - add: errno: -22 descr: Multi-Rail must be set to "True" or "False" found "Foobar" Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 99b (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 100: Add route with single gw (tcp) == 08:37:02 (1713530222) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp7 --gateway 192.168.202.15@tcp pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 100 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 101: Add route with multiple gw (tcp) ========================================================== 08:37:05 (1713530225) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp8 --gateway 192.168.202.[15-19]@tcp pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 101 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 102: Delete route with single gw ===== 08:37:10 (1713530230) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure