-----============= acceptance-small: conf-sanity ============----- Tue Apr 16 10:49:52 EDT 2024 excepting tests: 32newtarball 110 41c Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg411-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg411-server oleg411-server: oleg411-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg411-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg411-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg411-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:51:21 (1713279081) Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) oleg411-server: oleg411-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg411-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg411-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg411-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock oleg411-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5565 (time 1713279123.43 total 10.00 last 556.39) - open/close 10000 (time 1713279131.53 total 18.11 last 547.22) - open/close 15489 (time 1713279141.53 total 28.11 last 548.88) - open/close 20000 (time 1713279149.77 total 36.35 last 547.53) - open/close 25528 (time 1713279159.77 total 46.35 last 552.74) - open/close 30000 (time 1713279168.02 total 54.60 last 541.95) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.25 seconds: 549.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9733 (time 1713279183.48 total 10.00 last 973.25) - /unlink 19750 (time 1713279193.48 total 20.00 last 1001.63) - /unlink 29656 (time 1713279203.48 total 30.00 last 990.55) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.93 seconds: 988.63 ops/second Waiting for MDT destroys to complete - open/close 4093 (time 1713279248.92 total 10.00 last 409.24) - open/close 7811 (time 1713279258.92 total 20.00 last 371.80) - open/close 10000 (time 1713279264.22 total 25.30 last 413.14) - open/close 14038 (time 1713279274.22 total 35.30 last 403.78) - open/close 17183 (time 1713279284.22 total 45.30 last 314.44) - open/close 20000 (time 1713279292.34 total 53.42 last 347.12) - open/close 24876 (time 1713279302.34 total 63.42 last 487.52) - open/close 29643 (time 1713279312.34 total 73.42 last 476.65) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 80.43 seconds: 404.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7080 (time 1713279330.21 total 10.00 last 707.96) - /unlink 10000 (time 1713279334.18 total 13.96 last 736.58) - /unlink 18913 (time 1713279344.18 total 23.97 last 891.28) - /unlink 28714 (time 1713279354.18 total 33.97 last 980.04) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 38.35 seconds: 849.10 ops/second Waiting for MDT destroys to complete - open/close 4153 (time 1713279411.84 total 10.00 last 415.26) - open/close 7007 (time 1713279421.84 total 20.00 last 285.37) - open/close 10000 (time 1713279431.17 total 29.33 last 320.87) - open/close 15114 (time 1713279441.17 total 39.33 last 511.34) - open/close 20000 (time 1713279450.44 total 48.61 last 526.75) - open/close 24240 (time 1713279460.45 total 58.61 last 423.90) - open/close 28552 (time 1713279470.45 total 68.61 last 431.18) - open/close 30000 (time 1713279473.23 total 71.40 last 519.61) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 76.28 seconds: 426.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7603 (time 1713279489.13 total 10.00 last 760.29) - /unlink 10000 (time 1713279494.18 total 15.06 last 474.17) - /unlink 16787 (time 1713279504.18 total 25.06 last 678.70) - /unlink 20000 (time 1713279508.66 total 29.53 last 717.49) - /unlink 27716 (time 1713279518.66 total 39.53 last 771.57) - /unlink 30000 (time 1713279521.45 total 42.32 last 819.13) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 45.76 seconds: 711.57 ops/second Waiting for MDT destroys to complete - open/close 4970 (time 1713279551.96 total 10.00 last 496.94) - open/close 10000 (time 1713279561.94 total 19.98 last 503.93) - open/close 13919 (time 1713279571.94 total 29.99 last 391.80) - open/close 17930 (time 1713279581.94 total 39.99 last 401.07) - open/close 20000 (time 1713279586.23 total 44.28 last 482.52) - open/close 24412 (time 1713279596.24 total 54.28 last 441.16) - open/close 29733 (time 1713279606.24 total 64.28 last 532.02) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 69.32 seconds: 469.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9993 (time 1713279622.13 total 10.00 last 999.29) - /unlink 20000 (time 1713279632.12 total 19.99 last 1001.71) - /unlink 30000 (time 1713279642.01 total 29.88 last 1011.35) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.43 seconds: 1004.03 ops/second Waiting for MDT destroys to complete - open/close 4681 (time 1713279681.36 total 10.00 last 468.09) - open/close 9337 (time 1713279691.36 total 20.00 last 465.52) - open/close 14984 (time 1713279701.37 total 30.01 last 564.25) - open/close 20000 (time 1713279711.01 total 39.65 last 520.16) - open/close 25515 (time 1713279721.01 total 49.65 last 551.47) - open/close 29340 (time 1713279731.01 total 59.65 last 382.49) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 66.73 seconds: 487.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8324 (time 1713279748.92 total 10.00 last 832.25) - /unlink 10000 (time 1713279751.51 total 12.59 last 646.66) - /unlink 16775 (time 1713279761.52 total 22.59 last 677.45) - /unlink 20000 (time 1713279765.54 total 26.62 last 800.36) - /unlink 26625 (time 1713279775.55 total 36.62 last 662.44) - /unlink 30000 (time 1713279781.60 total 42.68 last 557.08) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 47.64 seconds: 683.45 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server unloading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 PASS 106 (738s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg411-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg411-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 oleg411-server: oleg411-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 851 sec =========== 11:04:04 (1713279844) === conf-sanity: start cleanup 11:04:05 (1713279845) === === conf-sanity: finish cleanup 11:04:05 (1713279845) ===