== conf-sanity test 106: check osp llog processing when catalog is wrapped =========================== 04:35:12 (1713515712) Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg313-server Starting mds1: lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: lustre-ost1/ost1 /mnt/lustre-ost1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1264 (time 1713515759.91 total 10.00 last 126.36) - open/close 2467 (time 1713515769.92 total 20.01 last 120.22) - open/close 3414 (time 1713515779.93 total 30.02 last 94.62) - open/close 4524 (time 1713515789.94 total 40.03 last 110.91) - open/close 5481 (time 1713515799.95 total 50.04 last 95.59) - open/close 6049 (time 1713515809.96 total 60.05 last 56.72) - open/close 6592 (time 1713515819.97 total 70.06 last 54.25) - open/close 7177 (time 1713515829.99 total 80.08 last 58.40) - open/close 7825 (time 1713515840.01 total 90.10 last 64.66) - open/close 8489 (time 1713515850.03 total 100.12 last 66.28) - open/close 9150 (time 1713515860.04 total 110.13 last 66.02) - open/close 9858 (time 1713515870.05 total 120.14 last 70.75) - open/close 10000 (time 1713515871.84 total 121.93 last 79.39) - open/close 10704 (time 1713515881.84 total 131.93 last 70.39) - open/close 11378 (time 1713515891.84 total 141.93 last 67.40) - open/close 12054 (time 1713515901.85 total 151.94 last 67.53) - open/close 12756 (time 1713515911.86 total 161.95 last 70.12) - open/close 13566 (time 1713515921.86 total 171.95 last 81.00) - open/close 14368 (time 1713515931.87 total 181.96 last 80.16) - open/close 15207 (time 1713515941.87 total 191.96 last 83.83) - open/close 16024 (time 1713515951.88 total 201.96 last 81.69) - open/close 17019 (time 1713515961.88 total 211.97 last 99.49) - open/close 18062 (time 1713515971.88 total 221.97 last 104.25) - open/close 19097 (time 1713515981.89 total 231.98 last 103.39) - open/close 20000 (time 1713515990.51 total 240.60 last 104.83) - open/close 20889 (time 1713516000.51 total 250.60 last 88.89) - open/close 21702 (time 1713516010.51 total 260.60 last 81.28) - open/close 22474 (time 1713516020.51 total 270.60 last 77.20) - open/close 23197 (time 1713516030.52 total 280.61 last 72.24) - open/close 23925 (time 1713516040.53 total 290.62 last 72.70) - open/close 24811 (time 1713516050.54 total 300.62 last 88.57) - open/close 25685 (time 1713516060.55 total 310.63 last 87.31) - open/close 26449 (time 1713516070.57 total 320.65 last 76.25) - open/close 27226 (time 1713516080.57 total 330.66 last 77.65) - open/close 27984 (time 1713516090.60 total 340.69 last 75.59) - open/close 28770 (time 1713516100.62 total 350.71 last 78.43) - open/close 29407 (time 1713516110.63 total 360.72 last 63.67) - open/close 30000 (time 1713516118.06 total 368.15 last 79.76) - open/close 30748 (time 1713516128.07 total 378.16 last 74.75) - open/close 31521 (time 1713516138.07 total 388.16 last 77.27) - open/close 32278 (time 1713516148.08 total 398.17 last 75.62) - open/close 33102 (time 1713516158.09 total 408.18 last 82.32) - open/close 33847 (time 1713516168.09 total 418.18 last 74.50) - open/close 34627 (time 1713516178.09 total 428.18 last 77.99) - open/close 35401 (time 1713516188.10 total 438.18 last 77.38) - open/close 36204 (time 1713516198.10 total 448.19 last 80.23) - open/close 36968 (time 1713516208.11 total 458.20 last 76.32) - open/close 37750 (time 1713516218.13 total 468.22 last 78.07) - open/close 38583 (time 1713516228.14 total 478.23 last 83.21) - open/close 39386 (time 1713516238.15 total 488.23 last 80.27) - open/close 40000 (time 1713516245.42 total 495.50 last 84.47) - open/close 40840 (time 1713516255.42 total 505.51 last 83.95) - open/close 41725 (time 1713516265.42 total 515.51 last 88.50) - open/close 42704 (time 1713516275.43 total 525.52 last 97.84) - open/close 43641 (time 1713516285.43 total 535.52 last 93.65) - open/close 44720 (time 1713516295.44 total 545.53 last 107.77) - open/close 45736 (time 1713516305.45 total 555.54 last 101.55) - open/close 46717 (time 1713516315.45 total 565.54 last 98.06) - open/close 47807 (time 1713516325.46 total 575.55 last 108.93) - open/close 48849 (time 1713516335.47 total 585.56 last 104.09) - open/close 49819 (time 1713516345.47 total 595.56 last 96.98) - open/close 50000 (time 1713516347.34 total 597.43 last 96.74) - open/close 50944 (time 1713516357.35 total 607.44 last 94.37) - open/close 51807 (time 1713516367.35 total 617.44 last 86.23) - open/close 52880 (time 1713516377.35 total 627.44 last 107.30) - open/close 54046 (time 1713516387.36 total 637.44 last 116.59) - open/close 55312 (time 1713516397.36 total 647.44 last 126.60) - open/close 56693 (time 1713516407.36 total 657.45 last 138.07) - open/close 57959 (time 1713516417.37 total 667.46 last 126.48) - open/close 59142 (time 1713516427.37 total 677.46 last 118.29) - open/close 60000 (time 1713516434.92 total 685.01 last 113.62) - open/close 61224 (time 1713516444.92 total 695.01 last 122.35) - open/close 62308 (time 1713516454.92 total 705.01 last 108.40) - open/close 63372 (time 1713516464.93 total 715.02 last 106.33) - open/close 64349 (time 1713516474.94 total 725.02 last 97.64) total: 64768 open/close in 729.48 seconds: 88.79 ops/second - /unlink 1835 (time 1713516489.41 total 10.01 last 183.31) - /unlink 3545 (time 1713516499.42 total 20.02 last 170.86) - /unlink 5270 (time 1713516509.42 total 30.02 last 172.45) - /unlink 7192 (time 1713516519.43 total 40.03 last 192.05) - /unlink 8893 (time 1713516529.45 total 50.05 last 169.80) - /unlink 9964 (time 1713516539.45 total 60.05 last 107.05) - /unlink 10000 (time 1713516539.92 total 60.52 last 77.49) - /unlink 11499 (time 1713516549.92 total 70.52 last 149.89) - /unlink 13454 (time 1713516559.92 total 80.52 last 195.44) - /unlink 15235 (time 1713516569.92 total 90.52 last 178.07) - /unlink 16716 (time 1713516579.93 total 100.53 last 147.97) - /unlink 17949 (time 1713516589.95 total 110.55 last 123.02) - /unlink 19314 (time 1713516599.96 total 120.57 last 136.33) - /unlink 20000 (time 1713516604.92 total 125.52 last 138.46) - /unlink 21568 (time 1713516614.93 total 135.53 last 156.69) - /unlink 23604 (time 1713516624.93 total 145.53 last 203.52) - /unlink 25711 (time 1713516634.93 total 155.53 last 210.70) - /unlink 27462 (time 1713516644.93 total 165.53 last 175.06) - /unlink 29172 (time 1713516654.94 total 175.54 last 170.92) - /unlink 30000 (time 1713516658.92 total 179.52 last 207.74) - /unlink 32536 (time 1713516668.93 total 189.53 last 253.53) - /unlink 34943 (time 1713516678.93 total 199.53 last 240.63) - /unlink 37159 (time 1713516688.93 total 209.53 last 221.54) - /unlink 39588 (time 1713516698.93 total 219.53 last 242.89) - /unlink 40000 (time 1713516700.62 total 221.22 last 243.47) - /unlink 42524 (time 1713516710.63 total 231.23 last 252.36) - /unlink 44847 (time 1713516720.63 total 241.23 last 232.25) - /unlink 47046 (time 1713516730.63 total 251.23 last 219.83) - /unlink 48958 (time 1713516740.63 total 261.23 last 191.16) - /unlink 50000 (time 1713516745.70 total 266.30 last 205.61) - /unlink 51835 (time 1713516755.71 total 276.31 last 183.37) - /unlink 53504 (time 1713516765.71 total 286.31 last 166.83) - /unlink 55497 (time 1713516775.71 total 296.31 last 199.27) - /unlink 57617 (time 1713516785.72 total 306.32 last 211.93) - /unlink 59550 (time 1713516795.72 total 316.32 last 193.26) - /unlink 60000 (time 1713516797.59 total 318.19 last 240.83) - /unlink 62136 (time 1713516807.59 total 328.19 last 213.58) - /unlink 64184 (time 1713516817.59 total 338.19 last 204.75) total: 64768 /unlink in 340.95 seconds: 189.96 ops/second - open/close 1332 (time 1713516830.36 total 10.00 last 133.15) - open/close 2800 (time 1713516840.36 total 20.01 last 146.76) - open/close 4341 (time 1713516850.36 total 30.01 last 154.01) - open/close 5845 (time 1713516860.36 total 40.01 last 150.39) - open/close 7441 (time 1713516870.37 total 50.02 last 159.53) - open/close 9109 (time 1713516880.37 total 60.02 last 166.76) - open/close 10000 (time 1713516885.11 total 64.76 last 188.04) - open/close 11874 (time 1713516895.11 total 74.76 last 187.37) - open/close 13692 (time 1713516905.11 total 84.76 last 181.79) - open/close 15636 (time 1713516915.11 total 94.76 last 194.39) - open/close 17722 (time 1713516925.12 total 104.76 last 208.51) - open/close 19647 (time 1713516935.12 total 114.76 last 192.47) - open/close 20000 (time 1713516936.77 total 116.41 last 213.97) - open/close 22009 (time 1713516946.77 total 126.42 last 200.88) - open/close 24105 (time 1713516956.77 total 136.42 last 209.53) - open/close 26124 (time 1713516966.77 total 146.42 last 201.88) - open/close 28012 (time 1713516976.78 total 156.42 last 188.71) - open/close 29950 (time 1713516986.78 total 166.43 last 193.73) - open/close 30000 (time 1713516987.01 total 166.66 last 218.53) - open/close 31916 (time 1713516997.01 total 176.66 last 191.50) - open/close 33787 (time 1713517007.02 total 186.66 last 187.07) - open/close 35869 (time 1713517017.02 total 196.66 last 208.18) - open/close 37984 (time 1713517027.02 total 206.67 last 211.45) - open/close 40000 (time 1713517036.16 total 215.81 last 220.51) - open/close 42138 (time 1713517046.17 total 225.81 last 213.70) - open/close 44188 (time 1713517056.17 total 235.82 last 204.94) - open/close 46273 (time 1713517066.17 total 245.82 last 208.46) - open/close 48400 (time 1713517076.17 total 255.82 last 212.62) - open/close 50000 (time 1713517084.90 total 264.54 last 183.44) - open/close 51952 (time 1713517094.90 total 274.55 last 195.14) - open/close 53989 (time 1713517104.90 total 284.55 last 203.67) - open/close 56050 (time 1713517114.90 total 294.55 last 206.03) - open/close 57647 (time 1713517124.91 total 304.56 last 159.64) - open/close 59090 (time 1713517134.91 total 314.56 last 144.30) - open/close 60000 (time 1713517140.81 total 320.45 last 154.28) - open/close 61714 (time 1713517150.81 total 330.46 last 171.36) - open/close 63673 (time 1713517160.81 total 340.46 last 195.86) total: 64768 open/close in 346.36 seconds: 187.00 ops/second - /unlink 2865 (time 1713517176.71 total 10.00 last 286.49) - /unlink 5860 (time 1713517186.71 total 20.00 last 299.45) - /unlink 9013 (time 1713517196.72 total 30.00 last 315.23) - /unlink 10000 (time 1713517199.43 total 32.72 last 363.69) - /unlink 13381 (time 1713517209.43 total 42.72 last 337.97) - /unlink 16484 (time 1713517219.44 total 52.73 last 310.18) - /unlink 19624 (time 1713517229.44 total 62.73 last 313.81) - /unlink 20000 (time 1713517230.97 total 64.26 last 246.44) - /unlink 23297 (time 1713517240.97 total 74.26 last 329.69) - /unlink 26671 (time 1713517250.97 total 84.26 last 337.34) - /unlink 30000 (time 1713517260.74 total 94.03 last 340.84) - /unlink 33515 (time 1713517270.74 total 104.03 last 351.47) - /unlink 37357 (time 1713517280.74 total 114.03 last 384.15) - /unlink 40000 (time 1713517288.54 total 121.83 last 338.89) - /unlink 43265 (time 1713517298.54 total 131.83 last 326.50) - /unlink 46781 (time 1713517308.54 total 141.83 last 351.52) - /unlink 50000 (time 1713517317.30 total 150.59 last 367.58) - /unlink 53557 (time 1713517327.30 total 160.59 last 355.68) - /unlink 57415 (time 1713517337.30 total 170.59 last 385.78) - /unlink 60000 (time 1713517345.49 total 178.78 last 315.76) - /unlink 63798 (time 1713517355.49 total 188.78 last 379.77) total: 64768 /unlink in 191.09 seconds: 338.95 ops/second - open/close 2140 (time 1713517367.80 total 10.00 last 213.99) - open/close 4292 (time 1713517377.80 total 20.00 last 215.13) - open/close 6288 (time 1713517387.80 total 30.01 last 199.56) - open/close 8460 (time 1713517397.81 total 40.01 last 217.16) - open/close 10000 (time 1713517404.69 total 46.89 last 223.62) - open/close 12378 (time 1713517414.70 total 56.90 last 237.72) - open/close 14453 (time 1713517424.70 total 66.90 last 207.45) - open/close 16705 (time 1713517434.70 total 76.91 last 225.09) - open/close 18927 (time 1713517444.71 total 86.91 last 222.12) - open/close 20000 (time 1713517449.57 total 91.77 last 220.71) - open/close 22185 (time 1713517459.57 total 101.77 last 218.46) - open/close 24182 (time 1713517469.57 total 111.77 last 199.69) - open/close 25846 (time 1713517479.58 total 121.78 last 166.27) - open/close 26876 (time 1713517489.58 total 131.78 last 103.00) - open/close 28288 (time 1713517499.58 total 141.78 last 141.16) - open/close 29834 (time 1713517509.58 total 151.78 last 154.60) - open/close 30000 (time 1713517510.46 total 152.66 last 189.82) - open/close 31538 (time 1713517520.46 total 162.66 last 153.76) - open/close 32922 (time 1713517530.47 total 172.67 last 138.33) - open/close 34258 (time 1713517540.47 total 182.67 last 133.57) - open/close 35597 (time 1713517550.47 total 192.67 last 133.86) - open/close 36958 (time 1713517560.47 total 202.68 last 136.05) - open/close 38300 (time 1713517570.48 total 212.68 last 134.14) - open/close 39642 (time 1713517580.48 total 222.68 last 134.19) - open/close 40000 (time 1713517583.72 total 225.92 last 110.52) - open/close 41440 (time 1713517593.72 total 235.92 last 143.97) - open/close 42804 (time 1713517603.72 total 245.93 last 136.35) - open/close 44442 (time 1713517613.73 total 255.93 last 163.74) - open/close 45949 (time 1713517623.73 total 265.93 last 150.67) - open/close 47331 (time 1713517633.73 total 275.93 last 138.16) - open/close 48731 (time 1713517643.74 total 285.94 last 139.91) - open/close 50000 (time 1713517651.07 total 293.27 last 173.15) - open/close 51941 (time 1713517661.07 total 303.27 last 194.04) - open/close 53876 (time 1713517671.08 total 313.28 last 193.39) - open/close 55618 (time 1713517681.08 total 323.28 last 174.17) - open/close 57147 (time 1713517691.08 total 333.29 last 152.81) - open/close 58830 (time 1713517701.09 total 343.29 last 168.18) - open/close 59704 (time 1713517711.10 total 353.30 last 87.30) - open/close 60000 (time 1713517715.47 total 357.67 last 67.85) - open/close 60703 (time 1713517725.47 total 367.67 last 70.29) - open/close 61562 (time 1713517735.47 total 377.67 last 85.87) - open/close 62542 (time 1713517745.47 total 387.67 last 97.98) - open/close 63729 (time 1713517755.48 total 397.68 last 118.61) - open/close 64686 (time 1713517765.48 total 407.69 last 95.67) total: 64768 open/close in 408.47 seconds: 158.56 ops/second - /unlink 1496 (time 1713517776.28 total 10.00 last 149.54) - /unlink 3273 (time 1713517786.28 total 20.01 last 177.66) - /unlink 5192 (time 1713517796.28 total 30.01 last 191.83) - /unlink 7719 (time 1713517806.29 total 40.01 last 252.67) - /unlink 9558 (time 1713517816.29 total 50.01 last 183.83) - /unlink 10000 (time 1713517818.65 total 52.38 last 186.92) - /unlink 11576 (time 1713517828.66 total 62.38 last 157.52) - /unlink 12994 (time 1713517838.66 total 72.39 last 141.77) - /unlink 14245 (time 1713517848.66 total 82.39 last 125.10) - /unlink 15557 (time 1713517858.66 total 92.39 last 131.15) - /unlink 16761 (time 1713517868.67 total 102.39 last 120.39) - /unlink 17928 (time 1713517878.67 total 112.40 last 116.65) - /unlink 19171 (time 1713517888.67 total 122.40 last 124.25) - /unlink 20000 (time 1713517895.60 total 129.32 last 119.73) - /unlink 21050 (time 1713517905.61 total 139.33 last 104.93) - /unlink 22145 (time 1713517915.61 total 149.33 last 109.47) - /unlink 23250 (time 1713517925.61 total 159.34 last 110.44) - /unlink 24379 (time 1713517935.62 total 169.35 last 112.77) - /unlink 25820 (time 1713517945.63 total 179.35 last 144.05) - /unlink 27243 (time 1713517955.64 total 189.36 last 142.17) - /unlink 28614 (time 1713517965.64 total 199.36 last 137.09) - /unlink 30000 (time 1713517973.84 total 207.57 last 168.97) - /unlink 31390 (time 1713517983.85 total 217.57 last 138.91) - /unlink 32381 (time 1713517993.86 total 227.58 last 99.00) - /unlink 33523 (time 1713518003.86 total 237.59 last 114.16) - /unlink 34825 (time 1713518013.86 total 247.59 last 130.14) - /unlink 36043 (time 1713518023.87 total 257.60 last 121.71) - /unlink 37368 (time 1713518033.88 total 267.60 last 132.45) - /unlink 38655 (time 1713518043.88 total 277.60 last 128.69) - /unlink 40000 (time 1713518053.66 total 287.38 last 137.53) - /unlink 41331 (time 1713518063.66 total 297.38 last 133.09) - /unlink 42738 (time 1713518073.66 total 307.39 last 140.65) - /unlink 43787 (time 1713518083.68 total 317.41 last 104.68) - /unlink 44698 (time 1713518093.69 total 327.42 last 91.02) - /unlink 45906 (time 1713518103.69 total 337.42 last 120.77) - /unlink 47215 (time 1713518113.70 total 347.42 last 130.88) - /unlink 48282 (time 1713518123.71 total 357.44 last 106.53) - /unlink 49422 (time 1713518133.72 total 367.45 last 113.86) - /unlink 50000 (time 1713518139.13 total 372.86 last 106.90) - /unlink 51133 (time 1713518149.13 total 382.86 last 113.29) - /unlink 52517 (time 1713518159.13 total 392.86 last 138.37) - /unlink 53990 (time 1713518169.13 total 402.86 last 147.29) - /unlink 55636 (time 1713518179.14 total 412.86 last 164.52) - /unlink 57011 (time 1713518189.15 total 422.87 last 137.38) - /unlink 58164 (time 1713518199.15 total 432.87 last 115.28) - /unlink 59537 (time 1713518209.15 total 442.88 last 137.27) - /unlink 60000 (time 1713518212.44 total 446.17 last 140.58) - /unlink 61387 (time 1713518222.45 total 456.17 last 138.67) - /unlink 62648 (time 1713518232.45 total 466.17 last 126.06) - /unlink 64136 (time 1713518242.45 total 476.18 last 148.77) total: 64768 /unlink in 479.66 seconds: 135.03 ops/second - open/close 1099 (time 1713518255.95 total 10.01 last 109.78) - open/close 2190 (time 1713518265.95 total 20.01 last 109.07) - open/close 3157 (time 1713518275.95 total 30.02 last 96.68) - open/close 4271 (time 1713518285.95 total 40.02 last 111.37) - open/close 5738 (time 1713518295.96 total 50.02 last 146.62) - open/close 7049 (time 1713518305.96 total 60.03 last 131.06) - open/close 8481 (time 1713518315.97 total 70.03 last 143.14) - open/close 9716 (time 1713518325.98 total 80.04 last 123.37) - open/close 10000 (time 1713518329.07 total 83.13 last 91.85) - open/close 10863 (time 1713518339.07 total 93.14 last 86.28) - open/close 11554 (time 1713518349.08 total 103.14 last 69.05) - open/close 12469 (time 1713518359.09 total 113.15 last 91.42) - open/close 13341 (time 1713518369.09 total 123.15 last 87.19) - open/close 14274 (time 1713518379.10 total 133.16 last 93.24) - open/close 15117 (time 1713518389.11 total 143.17 last 84.19) - open/close 15863 (time 1713518399.12 total 153.19 last 74.48) - open/close 16733 (time 1713518409.13 total 163.20 last 86.95) - open/close 17661 (time 1713518419.13 total 173.20 last 92.78) - open/close 18831 (time 1713518429.14 total 183.21 last 116.90) - open/close 20000 (time 1713518438.94 total 193.01 last 119.26) - open/close 21339 (time 1713518448.95 total 203.01 last 133.84) - open/close 22804 (time 1713518458.95 total 213.01 last 146.50) - open/close 24458 (time 1713518468.95 total 223.02 last 165.34) - open/close 25820 (time 1713518478.96 total 233.02 last 136.15) - open/close 27344 (time 1713518488.96 total 243.02 last 152.36) - open/close 28803 (time 1713518498.96 total 253.03 last 145.80) - open/close 30000 (time 1713518507.82 total 261.89 last 135.13) - open/close 31948 (time 1713518517.83 total 271.89 last 194.73) - open/close 34155 (time 1713518527.83 total 281.89 last 220.64) - open/close 35976 (time 1713518537.83 total 291.90 last 182.06) - open/close 37740 (time 1713518547.83 total 301.90 last 176.35) - open/close 39495 (time 1713518557.84 total 311.90 last 175.43) - open/close 40000 (time 1713518560.58 total 314.64 last 184.22) - open/close 42099 (time 1713518570.58 total 324.65 last 209.86) - open/close 44313 (time 1713518580.58 total 334.65 last 221.32) - open/close 46442 (time 1713518590.58 total 344.65 last 212.90) - open/close 48458 (time 1713518600.59 total 354.65 last 201.57) - open/close 50000 (time 1713518608.31 total 362.37 last 199.76) - open/close 52264 (time 1713518618.31 total 372.37 last 226.31) - open/close 54429 (time 1713518628.31 total 382.38 last 216.48) - open/close 56583 (time 1713518638.31 total 392.38 last 215.33) - open/close 58745 (time 1713518648.32 total 402.38 last 216.13) - open/close 60000 (time 1713518655.12 total 409.18 last 184.56) - open/close 61872 (time 1713518665.12 total 419.18 last 187.17) - open/close 63917 (time 1713518675.12 total 429.19 last 204.45) total: 64768 open/close in 433.35 seconds: 149.46 ops/second - /unlink 3056 (time 1713518689.29 total 10.00 last 305.58) - /unlink 6183 (time 1713518699.29 total 20.00 last 312.68) - /unlink 9761 (time 1713518709.29 total 30.00 last 357.70) - /unlink 10000 (time 1713518709.89 total 30.61 last 396.43) - /unlink 13432 (time 1713518719.90 total 40.61 last 343.13) - /unlink 16852 (time 1713518729.90 total 50.61 last 341.90) - /unlink 20000 (time 1713518738.34 total 59.06 last 372.83) - /unlink 23251 (time 1713518748.34 total 69.06 last 325.01) - /unlink 26828 (time 1713518758.35 total 79.06 last 357.65) - /unlink 30000 (time 1713518767.61 total 88.32 last 342.51) - /unlink 33407 (time 1713518777.61 total 98.32 last 340.65) - /unlink 36924 (time 1713518787.61 total 108.32 last 351.65) - /unlink 40000 (time 1713518796.79 total 117.50 last 335.21) - /unlink 43239 (time 1713518806.79 total 127.50 last 323.90) - /unlink 46477 (time 1713518816.79 total 137.50 last 323.73) - /unlink 49791 (time 1713518826.79 total 147.50 last 331.33) - /unlink 50000 (time 1713518827.37 total 148.08 last 363.29) - /unlink 53386 (time 1713518837.37 total 158.08 last 338.51) - /unlink 56833 (time 1713518847.37 total 168.08 last 344.62) - /unlink 60000 (time 1713518856.17 total 176.88 last 359.92) - /unlink 63728 (time 1713518866.17 total 186.88 last 372.77) total: 64768 /unlink in 189.77 seconds: 341.31 ops/second - open/close 2189 (time 1713518879.06 total 10.00 last 218.81) - open/close 4441 (time 1713518889.06 total 20.01 last 225.18) - open/close 6671 (time 1713518899.06 total 30.01 last 222.93) - open/close 8951 (time 1713518909.06 total 40.01 last 227.97) - open/close 10000 (time 1713518913.64 total 44.58 last 229.35) - open/close 12268 (time 1713518923.64 total 54.59 last 226.74) - open/close 14502 (time 1713518933.65 total 64.59 last 223.32) - open/close 16751 (time 1713518943.65 total 74.59 last 224.85) - open/close 19037 (time 1713518953.65 total 84.59 last 228.58) - open/close 20000 (time 1713518957.91 total 88.86 last 225.86) - open/close 22246 (time 1713518967.91 total 98.86 last 224.53) - open/close 24549 (time 1713518977.92 total 108.86 last 230.23) - open/close 26836 (time 1713518987.92 total 118.87 last 228.62) - open/close 29059 (time 1713518997.92 total 128.87 last 222.26) - open/close 30000 (time 1713519003.56 total 134.51 last 166.81) - open/close 31830 (time 1713519013.57 total 144.51 last 182.94) - open/close 33717 (time 1713519023.57 total 154.51 last 188.68) - open/close 35988 (time 1713519033.57 total 164.51 last 227.10) - open/close 38218 (time 1713519043.57 total 174.52 last 222.96) - open/close 40000 (time 1713519051.43 total 182.37 last 226.85) - open/close 42130 (time 1713519061.43 total 192.38 last 212.91) - open/close 44201 (time 1713519071.43 total 202.38 last 207.03) - open/close 46460 (time 1713519081.44 total 212.38 last 225.84) - open/close 48662 (time 1713519091.52 total 222.47 last 218.38) - open/close 50000 (time 1713519097.29 total 228.23 last 232.01) - open/close 52246 (time 1713519107.29 total 238.23 last 224.56) - open/close 54484 (time 1713519117.29 total 248.23 last 223.78) - open/close 56658 (time 1713519127.29 total 258.24 last 217.39) - open/close 58868 (time 1713519137.29 total 268.24 last 220.93) - open/close 60000 (time 1713519142.42 total 273.36 last 220.95) - open/close 62292 (time 1713519152.42 total 283.37 last 229.12) - open/close 64537 (time 1713519162.42 total 293.37 last 224.43) total: 64768 open/close in 294.55 seconds: 219.88 ops/second - /unlink 2612 (time 1713519173.61 total 10.00 last 261.20) - /unlink 4406 (time 1713519183.62 total 20.00 last 179.33) - /unlink 7595 (time 1713519193.62 total 30.01 last 318.83) - /unlink 10000 (time 1713519200.64 total 37.02 last 342.69) - /unlink 13289 (time 1713519210.64 total 47.03 last 328.78) - /unlink 15968 (time 1713519220.64 total 57.03 last 267.88) - /unlink 18946 (time 1713519230.64 total 67.03 last 297.79) - /unlink 20000 (time 1713519233.60 total 69.98 last 356.68) - /unlink 23387 (time 1713519243.60 total 79.99 last 338.60) - /unlink 26480 (time 1713519253.60 total 89.99 last 309.26) - /unlink 30000 (time 1713519263.26 total 99.65 last 364.42) - /unlink 33384 (time 1713519273.26 total 109.65 last 338.28) - /unlink 36130 (time 1713519283.27 total 119.65 last 274.50) - /unlink 39023 (time 1713519293.27 total 129.66 last 289.26) - /unlink 40000 (time 1713519296.62 total 133.01 last 291.54) - /unlink 43065 (time 1713519306.62 total 143.01 last 306.48) - /unlink 46025 (time 1713519316.62 total 153.01 last 295.99) - /unlink 49348 (time 1713519326.62 total 163.01 last 332.25) - /unlink 50000 (time 1713519328.86 total 165.25 last 291.49) - /unlink 53244 (time 1713519338.86 total 175.25 last 324.38) - /unlink 56518 (time 1713519348.86 total 185.25 last 327.38) - /unlink 60000 (time 1713519358.40 total 194.79 last 364.81) - /unlink 63589 (time 1713519368.41 total 204.79 last 358.86) total: 64768 /unlink in 208.03 seconds: 311.35 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for local destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server unloading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local modules unloaded. Resetting fail_loc on all nodes...pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 done. 05:36:50 (1713519410) waiting for oleg313-server network 5 secs ... 05:36:50 (1713519410) network interface is UP