-----============= acceptance-small: conf-sanity ============----- Thu Apr 18 20:16:31 EDT 2024 excepting tests: 32newtarball 110 41c Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg425-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg425-server oleg425-server: oleg425-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg425-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg425-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg425-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg425-server: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg425-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg425-server: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg425-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:17:55 (1713485875) Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f) oleg425-server: oleg425-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg425-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg425-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg425-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg425-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock oleg425-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4799 (time 1713485913.54 total 10.00 last 479.85) - open/close 9509 (time 1713485923.54 total 20.00 last 470.98) - open/close 14436 (time 1713485933.54 total 30.00 last 492.60) - open/close 19320 (time 1713485943.54 total 40.00 last 488.33) - open/close 24509 (time 1713485953.54 total 50.01 last 518.89) - open/close 29663 (time 1713485963.54 total 60.01 last 515.32) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 65.25 seconds: 499.02 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9486 (time 1713485979.67 total 10.00 last 948.56) - /unlink 19421 (time 1713485989.67 total 20.00 last 993.46) - /unlink 29379 (time 1713485999.67 total 30.00 last 995.76) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.32 seconds: 977.20 ops/second Waiting for MDT destroys to complete - open/close 5090 (time 1713486044.34 total 10.00 last 508.96) - open/close 10000 (time 1713486054.04 total 19.70 last 506.11) - open/close 15343 (time 1713486064.05 total 29.70 last 534.20) - open/close 17133 (time 1713486074.05 total 39.71 last 178.95) - open/close 18512 (time 1713486084.05 total 49.71 last 137.84) - open/close 20000 (time 1713486092.40 total 58.06 last 178.25) - open/close 21981 (time 1713486102.40 total 68.06 last 198.05) - open/close 24907 (time 1713486112.41 total 78.07 last 292.47) - open/close 26784 (time 1713486122.41 total 88.07 last 187.67) - open/close 28645 (time 1713486132.41 total 98.07 last 186.07) - open/close 30000 (time 1713486138.09 total 103.75 last 238.68) - open/close 32428 (time 1713486148.09 total 113.75 last 242.77) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 114.17 seconds: 285.19 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5389 (time 1713486159.56 total 10.00 last 538.78) - /unlink 10000 (time 1713486167.99 total 18.43 last 547.31) - /unlink 18909 (time 1713486177.99 total 28.43 last 890.85) - /unlink 27096 (time 1713486187.99 total 38.43 last 818.60) - /unlink 30000 (time 1713486192.31 total 42.76 last 671.24) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 46.08 seconds: 706.66 ops/second Waiting for MDT destroys to complete - open/close 4232 (time 1713486228.30 total 10.00 last 423.20) - open/close 8384 (time 1713486238.30 total 20.00 last 415.12) - open/close 10000 (time 1713486241.51 total 23.21 last 503.45) - open/close 15918 (time 1713486251.51 total 33.21 last 591.79) - open/close 20000 (time 1713486259.78 total 41.48 last 493.76) - open/close 25396 (time 1713486269.78 total 51.48 last 539.40) - open/close 29330 (time 1713486279.78 total 61.48 last 393.40) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 68.04 seconds: 478.55 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7305 (time 1713486297.28 total 10.00 last 730.44) - /unlink 10000 (time 1713486300.11 total 12.83 last 951.04) - /unlink 18749 (time 1713486310.11 total 22.84 last 874.85) - /unlink 28652 (time 1713486320.11 total 32.84 last 990.22) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 36.31 seconds: 896.80 ops/second Waiting for MDT destroys to complete - open/close 4873 (time 1713486357.88 total 10.00 last 487.28) - open/close 9466 (time 1713486367.88 total 20.00 last 459.23) - open/close 14110 (time 1713486377.88 total 30.00 last 464.39) - open/close 18053 (time 1713486387.88 total 40.00 last 394.27) - open/close 20000 (time 1713486392.10 total 44.23 last 460.64) - open/close 24520 (time 1713486402.10 total 54.23 last 451.99) - open/close 30000 (time 1713486412.00 total 64.13 last 553.77) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 69.83 seconds: 466.25 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7666 (time 1713486428.73 total 10.00 last 766.56) - /unlink 10000 (time 1713486431.66 total 12.94 last 794.50) - /unlink 18464 (time 1713486441.66 total 22.94 last 846.33) - /unlink 29027 (time 1713486451.66 total 32.94 last 1056.25) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 36.33 seconds: 896.14 ops/second Waiting for MDT destroys to complete - open/close 5823 (time 1713486492.56 total 10.00 last 582.29) - open/close 10000 (time 1713486499.75 total 17.19 last 581.28) - open/close 15822 (time 1713486509.75 total 27.19 last 582.15) - open/close 20000 (time 1713486517.73 total 35.17 last 523.60) - open/close 25822 (time 1713486527.73 total 45.17 last 582.17) - open/close 30000 (time 1713486534.78 total 52.22 last 592.67) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 56.39 seconds: 577.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1713486549.00 total 9.22 last 1085.02) - /unlink 20000 (time 1713486557.85 total 18.06 last 1130.81) - /unlink 30000 (time 1713486566.82 total 27.03 last 1114.29) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 29.23 seconds: 1113.98 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server unloading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 PASS 106 (746s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg425-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg425-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 oleg425-server: oleg425-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg425-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 847 sec =========== 20:30:39 (1713486639) === conf-sanity: start cleanup 20:30:39 (1713486639) === === conf-sanity: finish cleanup 20:30:39 (1713486639) ===