== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:30:35 (1713364235) Stopping clients: oleg259-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg259-client.virtnet /mnt/lustre2 (opts:-f) oleg259-server: oleg259-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: detected 4 online CPUs by sysfs oleg259-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg259-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg259-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5367 (time 1713364268.62 total 10.00 last 536.67) - open/close 10000 (time 1713364277.48 total 18.86 last 522.96) - open/close 15304 (time 1713364287.48 total 28.86 last 530.34) - open/close 20000 (time 1713364296.33 total 37.71 last 530.98) - open/close 25171 (time 1713364306.33 total 47.71 last 517.07) - open/close 30000 (time 1713364315.62 total 57.00 last 519.48) - open/close 35085 (time 1713364325.62 total 67.00 last 508.47) - open/close 40000 (time 1713364335.31 total 76.69 last 507.24) - open/close 44940 (time 1713364345.31 total 86.69 last 493.97) - open/close 49615 (time 1713364355.31 total 96.69 last 467.44) - open/close 53987 (time 1713364365.32 total 106.69 last 437.15) - open/close 57816 (time 1713364375.32 total 116.70 last 382.83) - open/close 60000 (time 1713364380.26 total 121.64 last 441.70) - open/close 64605 (time 1713364390.26 total 131.64 last 460.48) total: 64768 open/close in 131.96 seconds: 490.83 ops/second - /unlink 7864 (time 1713364401.42 total 10.00 last 786.40) - /unlink 10000 (time 1713364404.10 total 12.67 last 799.22) - /unlink 17316 (time 1713364414.10 total 22.67 last 731.51) - /unlink 20000 (time 1713364418.33 total 26.91 last 634.15) - /unlink 26616 (time 1713364428.33 total 36.91 last 661.57) - /unlink 30000 (time 1713364433.24 total 41.82 last 688.98) - /unlink 36320 (time 1713364443.24 total 51.82 last 631.98) - /unlink 40000 (time 1713364447.82 total 56.40 last 804.13) - /unlink 44025 (time 1713364457.82 total 66.40 last 402.36) - /unlink 46890 (time 1713364467.82 total 76.40 last 286.45) - /unlink 50000 (time 1713364472.26 total 80.84 last 700.98) - /unlink 55245 (time 1713364482.26 total 90.84 last 524.47) - /unlink 60000 (time 1713364489.35 total 97.93 last 670.25) total: 64768 /unlink in 106.63 seconds: 607.40 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5122 (time 1713364522.03 total 10.00 last 512.11) - open/close 9803 (time 1713364532.03 total 20.00 last 468.02) - open/close 14091 (time 1713364542.03 total 30.00 last 428.80) - open/close 18348 (time 1713364552.03 total 40.00 last 425.65) - open/close 20000 (time 1713364555.36 total 43.33 last 496.06) - open/close 24970 (time 1713364565.36 total 53.34 last 496.92) - open/close 28824 (time 1713364575.36 total 63.34 last 385.30) - open/close 30000 (time 1713364578.30 total 66.28 last 399.99) - open/close 34393 (time 1713364588.30 total 76.28 last 439.28) - open/close 38043 (time 1713364598.30 total 86.28 last 364.97) - open/close 40000 (time 1713364604.32 total 92.29 last 325.42) - open/close 43780 (time 1713364614.32 total 102.30 last 377.94) - open/close 48052 (time 1713364624.32 total 112.30 last 427.13) - open/close 50000 (time 1713364628.73 total 116.71 last 441.41) - open/close 54018 (time 1713364638.74 total 126.71 last 401.69) - open/close 57931 (time 1713364648.74 total 136.71 last 391.30) - open/close 60000 (time 1713364653.69 total 141.67 last 417.78) - open/close 63892 (time 1713364663.69 total 151.67 last 389.15) total: 64768 open/close in 153.90 seconds: 420.85 ops/second - /unlink 7644 (time 1713364676.80 total 10.00 last 764.31) - /unlink 10000 (time 1713364680.40 total 13.60 last 655.39) - /unlink 16275 (time 1713364690.40 total 23.60 last 627.47) - /unlink 20000 (time 1713364696.96 total 30.16 last 567.48) - /unlink 24855 (time 1713364706.96 total 40.16 last 485.43) - /unlink 30000 (time 1713364714.25 total 47.45 last 706.24) - /unlink 37565 (time 1713364724.25 total 57.45 last 756.47) - /unlink 40000 (time 1713364727.22 total 60.42 last 819.14) - /unlink 48317 (time 1713364737.22 total 70.42 last 831.64) - /unlink 50000 (time 1713364739.30 total 72.50 last 809.65) - /unlink 57648 (time 1713364749.30 total 82.50 last 764.73) - /unlink 60000 (time 1713364752.81 total 86.01 last 670.30) total: 64768 /unlink in 92.74 seconds: 698.35 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4440 (time 1713364787.82 total 10.00 last 443.97) - open/close 8707 (time 1713364797.82 total 20.00 last 426.64) - open/close 10000 (time 1713364800.37 total 22.55 last 507.17) - open/close 15246 (time 1713364810.37 total 32.55 last 524.55) - open/close 19851 (time 1713364820.37 total 42.55 last 460.39) - open/close 24803 (time 1713364830.37 total 52.56 last 495.18) - open/close 30000 (time 1713364839.75 total 61.94 last 554.05) - open/close 35254 (time 1713364849.75 total 71.94 last 525.39) - open/close 40000 (time 1713364858.48 total 80.66 last 544.11) - open/close 44838 (time 1713364868.48 total 90.66 last 483.75) - open/close 49348 (time 1713364878.48 total 100.66 last 450.92) - open/close 53649 (time 1713364888.48 total 110.66 last 430.02) - open/close 57967 (time 1713364898.48 total 120.66 last 431.77) - open/close 60000 (time 1713364903.92 total 126.10 last 373.77) - open/close 63686 (time 1713364913.92 total 136.10 last 368.53) total: 64768 open/close in 139.05 seconds: 465.79 ops/second - /unlink 7256 (time 1713364927.77 total 10.00 last 725.56) - /unlink 10000 (time 1713364931.42 total 13.64 last 752.97) - /unlink 16824 (time 1713364941.42 total 23.65 last 682.32) - /unlink 20000 (time 1713364946.42 total 28.65 last 634.44) - /unlink 27104 (time 1713364956.42 total 38.65 last 710.35) - /unlink 30000 (time 1713364960.40 total 42.63 last 727.95) - /unlink 37723 (time 1713364970.40 total 52.63 last 772.30) - /unlink 40000 (time 1713364973.73 total 55.96 last 683.43) - /unlink 47189 (time 1713364983.74 total 65.96 last 718.84) - /unlink 50000 (time 1713364987.07 total 69.29 last 844.07) - /unlink 58527 (time 1713364997.07 total 79.29 last 852.64) total: 64768 /unlink in 87.04 seconds: 744.16 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5131 (time 1713365032.85 total 10.00 last 513.06) - open/close 10000 (time 1713365042.68 total 19.83 last 495.45) - open/close 14509 (time 1713365052.68 total 29.83 last 450.89) - open/close 19361 (time 1713365062.68 total 39.83 last 485.13) - open/close 24632 (time 1713365072.68 total 49.83 last 527.02) - open/close 29225 (time 1713365082.68 total 59.83 last 459.22) - open/close 34093 (time 1713365092.68 total 69.83 last 486.79) - open/close 39003 (time 1713365102.68 total 79.83 last 490.93) - open/close 40000 (time 1713365104.87 total 82.03 last 454.87) - open/close 45101 (time 1713365114.88 total 92.03 last 510.04) - open/close 50000 (time 1713365124.82 total 101.97 last 492.67) - open/close 54745 (time 1713365134.82 total 111.97 last 474.47) - open/close 59279 (time 1713365144.82 total 121.97 last 453.38) - open/close 64309 (time 1713365154.82 total 131.97 last 502.99) total: 64768 open/close in 132.93 seconds: 487.24 ops/second - /unlink 8560 (time 1713365166.59 total 10.00 last 855.95) - /unlink 17428 (time 1713365176.59 total 20.00 last 886.80) - /unlink 20000 (time 1713365179.48 total 22.89 last 889.70) - /unlink 28165 (time 1713365189.48 total 32.89 last 816.45) - /unlink 30000 (time 1713365191.54 total 34.95 last 890.71) - /unlink 39442 (time 1713365201.54 total 44.95 last 944.12) - /unlink 49146 (time 1713365211.54 total 54.95 last 970.37) - /unlink 58297 (time 1713365221.54 total 64.95 last 915.06) total: 64768 /unlink in 72.20 seconds: 897.06 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4774 (time 1713365263.26 total 10.00 last 477.32) - open/close 9698 (time 1713365273.26 total 20.00 last 492.37) - open/close 14938 (time 1713365283.26 total 30.00 last 523.96) - open/close 19823 (time 1713365293.26 total 40.00 last 488.44) - open/close 24752 (time 1713365303.26 total 50.00 last 492.88) - open/close 29762 (time 1713365313.27 total 60.01 last 500.95) - open/close 34916 (time 1713365323.27 total 70.01 last 515.32) - open/close 40000 (time 1713365333.23 total 79.97 last 510.52) - open/close 44932 (time 1713365343.25 total 89.99 last 491.89) - open/close 49842 (time 1713365353.25 total 99.99 last 490.99) - open/close 54903 (time 1713365363.25 total 109.99 last 506.02) - open/close 59972 (time 1713365373.26 total 120.00 last 506.84) total: 64768 open/close in 129.40 seconds: 500.52 ops/second - /unlink 8169 (time 1713365393.51 total 10.00 last 816.78) - /unlink 10000 (time 1713365395.62 total 12.11 last 867.03) - /unlink 18479 (time 1713365405.62 total 22.11 last 847.88) - /unlink 26579 (time 1713365415.63 total 32.12 last 809.86) - /unlink 30000 (time 1713365420.37 total 36.86 last 721.20) - /unlink 38113 (time 1713365430.37 total 46.86 last 811.27) - /unlink 40000 (time 1713365432.61 total 49.10 last 842.78) - /unlink 48662 (time 1713365442.61 total 59.10 last 866.18) - /unlink 57514 (time 1713365452.61 total 69.10 last 885.13) - /unlink 60000 (time 1713365455.34 total 71.83 last 911.67) total: 64768 /unlink in 77.15 seconds: 839.49 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server unloading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2