-----============= acceptance-small: conf-sanity ============----- Tue Apr 16 15:40:47 EDT 2024 excepting tests: 32newtarball 110 41c Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg117-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server oleg117-server: oleg117-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: detected 4 online CPUs by sysfs oleg117-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:42:09 (1713296529) Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) oleg117-server: oleg117-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: detected 4 online CPUs by sysfs oleg117-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4331 (time 1713296568.76 total 10.00 last 433.06) - open/close 9100 (time 1713296578.76 total 20.00 last 476.89) - open/close 10000 (time 1713296580.86 total 22.10 last 429.05) - open/close 15294 (time 1713296590.86 total 32.10 last 529.37) - open/close 20000 (time 1713296599.71 total 40.94 last 532.20) - open/close 26222 (time 1713296609.71 total 50.94 last 622.13) - open/close 30000 (time 1713296615.98 total 57.22 last 601.97) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.69 seconds: 527.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1713296630.97 total 9.70 last 1031.16) - /unlink 19787 (time 1713296640.97 total 19.70 last 978.66) - /unlink 29477 (time 1713296650.97 total 29.70 last 968.92) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.91 seconds: 989.46 ops/second Waiting for MDT destroys to complete - open/close 5193 (time 1713296711.18 total 10.00 last 519.27) - open/close 10000 (time 1713296720.15 total 18.97 last 536.21) - open/close 15206 (time 1713296730.15 total 28.97 last 520.50) - open/close 20000 (time 1713296739.22 total 38.04 last 528.29) - open/close 25419 (time 1713296749.23 total 48.04 last 541.83) - open/close 30000 (time 1713296757.88 total 56.70 last 529.31) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.41 seconds: 530.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9479 (time 1713296773.48 total 10.00 last 947.84) - /unlink 19379 (time 1713296783.48 total 20.00 last 989.93) - /unlink 28572 (time 1713296793.48 total 30.00 last 919.27) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.62 seconds: 940.46 ops/second Waiting for MDT destroys to complete - open/close 5592 (time 1713296838.88 total 10.00 last 559.12) - open/close 10000 (time 1713296846.71 total 17.83 last 562.77) - open/close 15736 (time 1713296856.72 total 27.83 last 573.58) - open/close 20000 (time 1713296864.21 total 35.33 last 569.05) - open/close 25574 (time 1713296874.21 total 45.33 last 557.31) - open/close 30000 (time 1713296882.16 total 53.28 last 556.76) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 58.00 seconds: 561.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9231 (time 1713296897.81 total 10.00 last 923.04) - /unlink 19097 (time 1713296907.81 total 20.00 last 986.60) - /unlink 29027 (time 1713296917.81 total 30.00 last 992.93) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.47 seconds: 972.95 ops/second Waiting for MDT destroys to complete - open/close 5359 (time 1713296959.57 total 10.00 last 535.87) - open/close 10000 (time 1713296968.09 total 18.52 last 544.78) - open/close 15515 (time 1713296978.09 total 28.52 last 551.41) - open/close 20000 (time 1713296986.51 total 36.94 last 533.00) - open/close 25416 (time 1713296996.51 total 46.94 last 541.55) - open/close 30000 (time 1713297004.80 total 55.23 last 553.02) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.88 seconds: 543.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32692 47300 41% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32947 32947 0 100% /mnt/lustre - /unlink 9729 (time 1713297020.32 total 10.00 last 972.89) - /unlink 18346 (time 1713297030.32 total 20.00 last 861.62) - /unlink 20000 (time 1713297032.34 total 22.01 last 822.32) - /unlink 29413 (time 1713297042.34 total 32.01 last 941.26) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 35.09 seconds: 927.83 ops/second Waiting for MDT destroys to complete - open/close 4867 (time 1713297081.08 total 10.00 last 486.64) - open/close 10000 (time 1713297090.71 total 19.62 last 533.48) - open/close 15197 (time 1713297100.71 total 29.62 last 519.67) - open/close 20000 (time 1713297110.21 total 39.13 last 505.39) - open/close 25635 (time 1713297120.21 total 49.13 last 563.47) - open/close 30000 (time 1713297128.64 total 57.56 last 517.50) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.11 seconds: 524.27 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1713297144.02 total 9.95 last 1005.25) - /unlink 20000 (time 1713297153.76 total 19.69 last 1026.62) - /unlink 29918 (time 1713297163.76 total 29.69 last 991.76) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.23 seconds: 1010.13 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server unloading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 PASS 106 (689s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg117-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg117-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 oleg117-server: oleg117-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: detected 4 online CPUs by sysfs oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 790 sec =========== 15:53:57 (1713297237) === conf-sanity: start cleanup 15:53:58 (1713297238) === === conf-sanity: finish cleanup 15:53:58 (1713297238) ===