== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:59:22 (1713293962) Stopping clients: oleg445-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg445-client.virtnet /mnt/lustre2 (opts:-f) oleg445-server: oleg445-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1738 (time 1713294003.23 total 10.00 last 173.79) - open/close 3362 (time 1713294013.23 total 20.00 last 162.36) - open/close 4900 (time 1713294023.24 total 30.01 last 153.71) - open/close 6767 (time 1713294033.24 total 40.01 last 186.62) - open/close 8564 (time 1713294043.24 total 50.01 last 179.70) - open/close 10000 (time 1713294051.52 total 58.29 last 173.43) - open/close 11738 (time 1713294061.53 total 68.30 last 173.76) - open/close 13377 (time 1713294071.53 total 78.30 last 163.87) - open/close 15081 (time 1713294081.53 total 88.30 last 170.36) - open/close 17383 (time 1713294091.54 total 98.31 last 230.06) - open/close 19329 (time 1713294101.54 total 108.31 last 194.59) - open/close 20000 (time 1713294105.17 total 111.94 last 184.53) - open/close 21863 (time 1713294115.18 total 121.95 last 186.24) - open/close 23454 (time 1713294125.18 total 131.95 last 159.09) - open/close 25374 (time 1713294135.18 total 141.95 last 191.96) - open/close 28857 (time 1713294145.18 total 151.95 last 348.24) - open/close 30000 (time 1713294148.23 total 155.00 last 374.20) - open/close 33272 (time 1713294158.24 total 165.01 last 327.16) - open/close 36433 (time 1713294168.24 total 175.01 last 316.07) - open/close 38811 (time 1713294178.24 total 185.01 last 237.74) - open/close 40000 (time 1713294181.48 total 188.25 last 366.58) - open/close 43979 (time 1713294191.48 total 198.25 last 397.81) - open/close 48483 (time 1713294201.49 total 208.26 last 450.34) - open/close 50000 (time 1713294204.30 total 211.07 last 538.16) - open/close 54245 (time 1713294214.31 total 221.08 last 424.42) - open/close 57934 (time 1713294224.31 total 231.08 last 368.84) - open/close 60000 (time 1713294229.85 total 236.62 last 372.97) - open/close 64600 (time 1713294239.85 total 246.62 last 460.00) total: 64768 open/close in 247.06 seconds: 262.15 ops/second - /unlink 7489 (time 1713294251.16 total 10.00 last 748.80) - /unlink 10000 (time 1713294255.14 total 13.99 last 629.97) - /unlink 16920 (time 1713294265.15 total 23.99 last 691.87) - /unlink 20000 (time 1713294271.16 total 30.00 last 512.42) - /unlink 24768 (time 1713294281.16 total 40.00 last 476.72) - /unlink 30000 (time 1713294290.94 total 49.78 last 535.07) - /unlink 34583 (time 1713294300.94 total 59.78 last 458.23) - /unlink 39052 (time 1713294310.94 total 69.78 last 446.83) - /unlink 44282 (time 1713294320.94 total 79.78 last 522.95) - /unlink 49523 (time 1713294330.94 total 89.79 last 524.00) - /unlink 53972 (time 1713294340.94 total 99.79 last 444.81) - /unlink 59001 (time 1713294350.95 total 109.79 last 502.86) - /unlink 64206 (time 1713294360.95 total 119.79 last 520.40) total: 64768 /unlink in 120.76 seconds: 536.33 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2359 (time 1713294390.13 total 10.00 last 235.90) - open/close 5570 (time 1713294400.13 total 20.00 last 321.01) - open/close 8102 (time 1713294410.13 total 30.01 last 253.15) - open/close 10000 (time 1713294417.52 total 37.39 last 257.09) - open/close 12919 (time 1713294427.52 total 47.39 last 291.88) - open/close 15573 (time 1713294437.52 total 57.39 last 265.39) - open/close 18186 (time 1713294447.52 total 67.39 last 261.23) - open/close 20000 (time 1713294454.01 total 73.88 last 279.42) - open/close 22578 (time 1713294464.02 total 83.89 last 257.74) - open/close 25290 (time 1713294474.02 total 93.89 last 271.17) - open/close 28454 (time 1713294484.02 total 103.89 last 316.31) - open/close 30000 (time 1713294488.47 total 108.34 last 347.60) - open/close 33997 (time 1713294498.47 total 118.34 last 399.63) - open/close 37620 (time 1713294508.47 total 128.34 last 362.26) - open/close 40000 (time 1713294514.12 total 133.99 last 421.31) - open/close 44244 (time 1713294524.12 total 143.99 last 424.34) - open/close 48475 (time 1713294534.12 total 153.99 last 423.01) - open/close 50000 (time 1713294537.60 total 157.47 last 439.15) - open/close 54080 (time 1713294547.60 total 167.47 last 407.92) - open/close 58339 (time 1713294557.60 total 177.47 last 425.80) - open/close 60000 (time 1713294562.26 total 182.13 last 356.62) - open/close 64044 (time 1713294572.26 total 192.13 last 404.31) total: 64768 open/close in 193.98 seconds: 333.89 ops/second - /unlink 6995 (time 1713294585.03 total 10.00 last 699.45) - /unlink 10000 (time 1713294588.93 total 13.90 last 770.67) - /unlink 18490 (time 1713294598.93 total 23.90 last 848.97) - /unlink 20000 (time 1713294601.05 total 26.02 last 712.44) - /unlink 27467 (time 1713294611.05 total 36.02 last 746.68) - /unlink 30000 (time 1713294614.18 total 39.15 last 809.65) - /unlink 38094 (time 1713294624.18 total 49.15 last 809.35) - /unlink 40000 (time 1713294626.69 total 51.66 last 759.82) - /unlink 48575 (time 1713294636.69 total 61.66 last 857.45) - /unlink 57069 (time 1713294646.69 total 71.66 last 849.28) - /unlink 60000 (time 1713294650.90 total 75.87 last 696.57) total: 64768 /unlink in 82.44 seconds: 785.59 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5029 (time 1713294685.10 total 10.00 last 502.90) - open/close 9580 (time 1713294695.10 total 20.00 last 455.06) - open/close 14276 (time 1713294705.10 total 30.00 last 469.58) - open/close 19311 (time 1713294715.11 total 40.00 last 503.43) - open/close 24289 (time 1713294725.11 total 50.00 last 497.78) - open/close 29523 (time 1713294735.11 total 60.00 last 523.39) - open/close 34290 (time 1713294745.11 total 70.00 last 476.67) - open/close 39849 (time 1713294755.11 total 80.01 last 555.84) - open/close 45119 (time 1713294765.11 total 90.01 last 526.98) - open/close 50000 (time 1713294774.62 total 99.51 last 513.34) - open/close 55330 (time 1713294784.62 total 109.52 last 532.88) - open/close 60000 (time 1713294793.92 total 118.81 last 502.33) total: 64768 open/close in 127.80 seconds: 506.79 ops/second - /unlink 8951 (time 1713294813.74 total 10.00 last 895.04) - /unlink 18037 (time 1713294823.74 total 20.00 last 908.53) - /unlink 20000 (time 1713294825.78 total 22.04 last 963.13) - /unlink 29523 (time 1713294835.78 total 32.04 last 952.28) - /unlink 39052 (time 1713294845.78 total 42.04 last 952.82) - /unlink 48747 (time 1713294855.78 total 52.04 last 969.41) - /unlink 58301 (time 1713294865.78 total 62.04 last 955.39) total: 64768 /unlink in 68.53 seconds: 945.11 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5561 (time 1713294911.58 total 10.00 last 556.02) - open/close 10000 (time 1713294919.30 total 17.72 last 574.81) - open/close 15619 (time 1713294929.31 total 27.73 last 561.80) - open/close 20000 (time 1713294937.33 total 35.75 last 545.83) - open/close 24787 (time 1713294947.33 total 45.75 last 478.69) - open/close 29572 (time 1713294957.33 total 55.75 last 478.45) - open/close 34395 (time 1713294967.34 total 65.75 last 482.23) - open/close 39251 (time 1713294977.34 total 75.76 last 485.57) - open/close 44272 (time 1713294987.34 total 85.76 last 502.02) - open/close 49383 (time 1713294997.34 total 95.76 last 511.06) - open/close 54441 (time 1713295007.34 total 105.76 last 505.77) - open/close 59604 (time 1713295017.34 total 115.76 last 516.22) total: 64768 open/close in 125.71 seconds: 515.21 ops/second - /unlink 8813 (time 1713295038.17 total 10.00 last 881.30) - /unlink 17469 (time 1713295048.17 total 20.00 last 865.57) - /unlink 20000 (time 1713295051.07 total 22.90 last 873.10) - /unlink 28558 (time 1713295061.07 total 32.90 last 855.77) - /unlink 30000 (time 1713295063.30 total 35.13 last 645.23) - /unlink 38556 (time 1713295073.30 total 45.13 last 855.59) - /unlink 46932 (time 1713295083.30 total 55.14 last 837.52) - /unlink 50000 (time 1713295086.73 total 58.56 last 894.74) - /unlink 59033 (time 1713295096.73 total 68.57 last 903.24) total: 64768 /unlink in 74.86 seconds: 865.18 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5382 (time 1713295136.66 total 10.00 last 538.11) - open/close 10000 (time 1713295145.33 total 18.68 last 532.40) - open/close 15293 (time 1713295155.33 total 28.68 last 529.20) - open/close 20000 (time 1713295164.25 total 37.59 last 527.86) - open/close 25379 (time 1713295174.25 total 47.60 last 537.83) - open/close 30000 (time 1713295183.09 total 56.43 last 522.92) - open/close 35094 (time 1713295193.09 total 66.43 last 509.35) - open/close 40000 (time 1713295202.59 total 75.93 last 516.58) - open/close 45129 (time 1713295212.59 total 85.93 last 512.81) - open/close 50000 (time 1713295222.00 total 95.34 last 517.65) - open/close 55053 (time 1713295232.00 total 105.34 last 505.21) - open/close 60000 (time 1713295241.85 total 115.20 last 501.96) total: 64768 open/close in 124.63 seconds: 519.67 ops/second - /unlink 8454 (time 1713295262.14 total 10.00 last 845.33) - /unlink 16878 (time 1713295272.14 total 20.00 last 842.38) - /unlink 20000 (time 1713295275.82 total 23.68 last 847.80) - /unlink 28698 (time 1713295285.82 total 33.68 last 869.70) - /unlink 37277 (time 1713295295.82 total 43.68 last 857.90) - /unlink 40000 (time 1713295298.91 total 46.77 last 882.45) - /unlink 48851 (time 1713295308.91 total 56.77 last 885.06) - /unlink 57777 (time 1713295318.91 total 66.77 last 892.52) - /unlink 60000 (time 1713295321.47 total 69.33 last 869.39) total: 64768 /unlink in 74.74 seconds: 866.63 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2