== replay-dual test 26: dbench and tar with mds failover ========================================================== 09:28:18 (1713533298) Starting client oleg305-client.virtnet: -o user_xattr,flock oleg305-server@tcp:/lustre /mnt/lustre Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 6396 Started dbench loop with 6397 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Apr 19 09:28:20 EDT 2024 waiting for dbench pid 6438 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 203 7.91 MB/sec warmup 1 sec latency 27.332 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26392 3555624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28188 7160848 1% /mnt/lustre 1 404 7.17 MB/sec warmup 2 sec latency 28.710 ms 1 659 7.04 MB/sec warmup 3 sec latency 150.577 ms 1 983 5.48 MB/sec warmup 4 sec latency 20.670 ms 1 1259 4.62 MB/sec warmup 5 sec latency 26.140 ms test_26 fail mds1 1 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 1653 4.37 MB/sec warmup 6 sec latency 24.491 ms 1 1745 3.75 MB/sec warmup 7 sec latency 694.554 ms 09:28:27 (1713533307) shut down 1 1745 3.28 MB/sec warmup 8 sec latency 1694.822 ms 1 1745 2.92 MB/sec warmup 9 sec latency 2695.077 ms 1 1745 2.63 MB/sec warmup 10 sec latency 3695.323 ms 1 1745 2.39 MB/sec warmup 11 sec latency 4695.586 ms 1 1745 2.19 MB/sec warmup 12 sec latency 5695.844 ms 1 1745 2.02 MB/sec warmup 13 sec latency 6696.099 ms 1 1745 1.88 MB/sec warmup 14 sec latency 7696.502 ms 1 1745 1.75 MB/sec warmup 15 sec latency 8696.774 ms 1 1745 1.64 MB/sec warmup 16 sec latency 9697.061 ms 1 1745 1.55 MB/sec warmup 17 sec latency 10697.493 ms Failover mds1 to oleg305-server mount facets: mds1 1 1745 1.46 MB/sec warmup 18 sec latency 11697.753 ms 1 1745 1.38 MB/sec warmup 19 sec latency 12697.967 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1745 0.00 MB/sec execute 1 sec latency 14698.363 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 1745 0.00 MB/sec execute 2 sec latency 15698.643 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 09:28:43 (1713533323) targets are mounted 09:28:43 (1713533323) facet_failover done 1 1745 0.00 MB/sec execute 3 sec latency 16698.935 ms 1 1745 0.00 MB/sec execute 4 sec latency 17699.150 ms 1 1745 0.00 MB/sec execute 5 sec latency 18699.483 ms 1 1745 0.00 MB/sec execute 6 sec latency 19699.726 ms 1 1745 0.00 MB/sec execute 7 sec latency 20699.944 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2039 0.04 MB/sec execute 8 sec latency 20811.217 ms 1 2391 0.19 MB/sec execute 9 sec latency 22.182 ms 1 2602 0.27 MB/sec execute 10 sec latency 23.915 ms 1 3010 0.66 MB/sec execute 11 sec latency 23.493 ms 1 3432 0.85 MB/sec execute 12 sec latency 24.776 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2908 1284780 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2600 1285088 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 32264 3565908 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6304 3589748 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38568 7155656 1% /mnt/lustre 1 3785 1.04 MB/sec execute 13 sec latency 25.456 ms 1 3933 0.98 MB/sec execute 14 sec latency 25.078 ms 1 4101 0.92 MB/sec execute 15 sec latency 25.288 ms 1 4312 0.88 MB/sec execute 16 sec latency 39.151 ms test_26 fail mds2 2 times 1 4597 0.92 MB/sec execute 17 sec latency 23.636 ms Failing mds2 on oleg305-server Stopping /mnt/lustre-mds2 (opts:) on oleg305-server 1 4828 0.89 MB/sec execute 18 sec latency 43.310 ms 09:28:59 (1713533339) shut down 1 5216 1.00 MB/sec execute 19 sec latency 26.610 ms 1 5666 0.98 MB/sec execute 20 sec latency 28.136 ms 1 5852 0.98 MB/sec execute 21 sec latency 658.990 ms 1 5852 0.94 MB/sec execute 22 sec latency 1659.114 ms 1 5852 0.90 MB/sec execute 23 sec latency 2659.269 ms 1 5852 0.86 MB/sec execute 24 sec latency 3659.446 ms 1 5852 0.82 MB/sec execute 25 sec latency 4659.587 ms 1 5852 0.79 MB/sec execute 26 sec latency 5659.702 ms 1 5852 0.76 MB/sec execute 27 sec latency 6659.861 ms 1 5852 0.74 MB/sec execute 28 sec latency 7659.966 ms Failover mds2 to oleg305-server mount facets: mds2 1 5852 0.71 MB/sec execute 29 sec latency 8660.060 ms 1 5852 0.69 MB/sec execute 30 sec latency 9660.149 ms 1 5852 0.66 MB/sec execute 31 sec latency 10660.250 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 5852 0.64 MB/sec execute 32 sec latency 11660.373 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 5852 0.62 MB/sec execute 33 sec latency 12660.702 ms Started lustre-MDT0001 09:29:13 (1713533353) targets are mounted 09:29:13 (1713533353) facet_failover done 1 5852 0.61 MB/sec execute 34 sec latency 13660.902 ms 1 5852 0.59 MB/sec execute 35 sec latency 14661.087 ms 1 5852 0.57 MB/sec execute 36 sec latency 15661.283 ms 1 5852 0.56 MB/sec execute 37 sec latency 16661.499 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 5986 0.55 MB/sec execute 38 sec latency 16966.578 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6317 0.64 MB/sec execute 39 sec latency 24.585 ms 1 6726 0.67 MB/sec execute 40 sec latency 23.840 ms 1 7084 0.76 MB/sec execute 41 sec latency 24.059 ms 1 7341 0.78 MB/sec execute 42 sec latency 40.089 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4116 1283572 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37532 3550636 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12704 3577152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50236 7127788 1% /mnt/lustre 1 7488 0.77 MB/sec execute 43 sec latency 24.838 ms 1 7656 0.75 MB/sec execute 44 sec latency 23.264 ms 1 7883 0.74 MB/sec execute 45 sec latency 149.151 ms 1 8161 0.76 MB/sec execute 46 sec latency 24.085 ms test_26 fail mds1 3 times 1 8597 0.81 MB/sec execute 47 sec latency 27.374 ms Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 9142 0.80 MB/sec execute 48 sec latency 20.115 ms 09:29:29 (1713533369) shut down 1 9194 0.79 MB/sec execute 49 sec latency 890.025 ms 1 9194 0.77 MB/sec execute 50 sec latency 1890.185 ms 1 9194 0.76 MB/sec execute 51 sec latency 2890.356 ms 1 9194 0.74 MB/sec execute 52 sec latency 3890.549 ms 1 9194 0.73 MB/sec execute 53 sec latency 4890.714 ms 1 9194 0.71 MB/sec execute 54 sec latency 5890.864 ms 1 9194 0.70 MB/sec execute 55 sec latency 6891.005 ms 1 9194 0.69 MB/sec execute 56 sec latency 7891.169 ms 1 9194 0.68 MB/sec execute 57 sec latency 8891.330 ms 1 9194 0.67 MB/sec execute 58 sec latency 9891.510 ms Failover mds1 to oleg305-server 1 9194 0.65 MB/sec execute 59 sec latency 10891.602 ms mount facets: mds1 1 9194 0.64 MB/sec execute 60 sec latency 11891.742 ms 1 9194 0.63 MB/sec execute 61 sec latency 12891.927 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 9194 0.62 MB/sec execute 62 sec latency 13892.127 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 09:29:43 (1713533383) targets are mounted 09:29:43 (1713533383) facet_failover done 1 9194 0.61 MB/sec execute 63 sec latency 14892.352 ms 1 9194 0.60 MB/sec execute 64 sec latency 15892.534 ms 1 9194 0.59 MB/sec execute 65 sec latency 16892.719 ms 1 9194 0.58 MB/sec execute 66 sec latency 17892.890 ms 1 9194 0.58 MB/sec execute 67 sec latency 18893.049 ms 1 9194 0.57 MB/sec execute 68 sec latency 19893.317 ms 1 9194 0.56 MB/sec execute 69 sec latency 20893.547 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 9665 0.58 MB/sec execute 70 sec latency 20932.442 ms 1 10384 0.67 MB/sec execute 71 sec latency 19.200 ms 1 10891 0.72 MB/sec execute 72 sec latency 20.471 ms 1 11118 0.71 MB/sec execute 73 sec latency 37.665 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4548 1283140 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2588 1285100 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37808 3548048 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13100 3574884 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50908 7122932 1% /mnt/lustre 1 11471 0.71 MB/sec execute 74 sec latency 17.315 ms 1 11901 0.72 MB/sec execute 75 sec latency 15.026 ms 1 12466 0.76 MB/sec execute 76 sec latency 21.965 ms 1 13031 0.77 MB/sec execute 77 sec latency 15.024 ms test_26 fail mds2 4 times Failing mds2 on oleg305-server Stopping /mnt/lustre-mds2 (opts:) on oleg305-server 1 13532 0.81 MB/sec execute 78 sec latency 23.556 ms 09:29:59 (1713533399) shut down 1 13632 0.82 MB/sec execute 79 sec latency 792.205 ms 1 13632 0.81 MB/sec execute 80 sec latency 1792.411 ms 1 13632 0.80 MB/sec execute 81 sec latency 2792.691 ms 1 13632 0.79 MB/sec execute 82 sec latency 3793.010 ms 1 13632 0.78 MB/sec execute 83 sec latency 4793.260 ms 1 13632 0.77 MB/sec execute 84 sec latency 5793.567 ms 1 13632 0.76 MB/sec execute 85 sec latency 6793.865 ms 1 13632 0.75 MB/sec execute 86 sec latency 7794.069 ms 1 13632 0.74 MB/sec execute 87 sec latency 8794.359 ms 1 13632 0.73 MB/sec execute 88 sec latency 9794.619 ms Failover mds2 to oleg305-server mount facets: mds2 1 13632 0.73 MB/sec execute 89 sec latency 10794.792 ms 1 13632 0.72 MB/sec execute 90 sec latency 11794.983 ms 1 13632 0.71 MB/sec execute 91 sec latency 12795.244 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 13632 0.70 MB/sec execute 92 sec latency 13795.493 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 13632 0.69 MB/sec execute 93 sec latency 14795.675 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0001 09:30:14 (1713533414) targets are mounted 09:30:14 (1713533414) facet_failover done 1 13632 0.69 MB/sec execute 94 sec latency 15795.928 ms 1 13632 0.68 MB/sec execute 95 sec latency 16796.232 ms 1 13632 0.67 MB/sec execute 96 sec latency 17796.563 ms 1 13632 0.67 MB/sec execute 97 sec latency 18796.781 ms 1 13646 0.66 MB/sec execute 98 sec latency 19710.597 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 14270 0.70 MB/sec execute 99 sec latency 14.060 ms 1 cleanup 100 sec 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2152 6.557 40.060 Close 1598 13.882 20811.202 Rename 93 15.413 28.117 Unlink 424 6.872 36.435 Qpathinfo 1999 22.041 20932.427 Qfileinfo 354 0.536 3.108 Qfsinfo 367 0.998 7.380 Sfileinfo 166 10.405 43.297 Find 775 2.434 16.971 WriteX 1092 2.650 12.525 ReadX 3569 0.102 10.782 LockX 8 2.122 3.169 UnlockX 8 2.231 3.327 Flush 145 147.884 19710.586 Throughput 0.703731 MB/sec 1 clients 1 procs max_latency=20932.442 ms stopping dbench on /mnt/lustre at Fri Apr 19 09:30:21 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5108 1282580 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3196 1284492 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45176 3548288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7740 3595540 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 52916 7143828 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Apr 19 09:30:23 EDT 2024 waiting for dbench pid 10916 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 120 5.50 MB/sec warmup 1 sec latency 364.776 ms 1 296 5.48 MB/sec warmup 2 sec latency 517.207 ms 1 448 5.21 MB/sec warmup 3 sec latency 29.359 ms test_26 fail mds1 5 times Failing mds1 on oleg305-server 1 728 5.33 MB/sec warmup 4 sec latency 32.274 ms Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 869 4.33 MB/sec warmup 5 sec latency 448.372 ms 1 869 3.61 MB/sec warmup 6 sec latency 1448.594 ms 1 869 3.09 MB/sec warmup 7 sec latency 2448.754 ms 1 869 2.71 MB/sec warmup 8 sec latency 3448.973 ms 1 869 2.41 MB/sec warmup 9 sec latency 4449.229 ms 1 869 2.16 MB/sec warmup 10 sec latency 5449.427 ms 09:30:34 (1713533434) shut down 1 869 1.97 MB/sec warmup 11 sec latency 6449.607 ms 1 869 1.80 MB/sec warmup 12 sec latency 7449.797 ms 1 869 1.67 MB/sec warmup 13 sec latency 8449.998 ms 1 869 1.55 MB/sec warmup 14 sec latency 9450.270 ms 1 869 1.44 MB/sec warmup 15 sec latency 10450.571 ms 1 869 1.35 MB/sec warmup 16 sec latency 11450.799 ms 1 869 1.27 MB/sec warmup 17 sec latency 12451.066 ms 1 869 1.20 MB/sec warmup 18 sec latency 13451.327 ms 1 869 1.14 MB/sec warmup 19 sec latency 14451.507 ms Failover mds1 to oleg305-server mount facets: mds1 1 869 0.00 MB/sec execute 1 sec latency 16451.802 ms 1 869 0.00 MB/sec execute 2 sec latency 17451.936 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 869 0.00 MB/sec execute 3 sec latency 18452.061 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 869 0.00 MB/sec execute 4 sec latency 19452.247 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 09:30:48 (1713533448) targets are mounted 09:30:48 (1713533448) facet_failover done 1 869 0.00 MB/sec execute 5 sec latency 20452.539 ms 1 869 0.00 MB/sec execute 6 sec latency 21452.748 ms 1 869 0.00 MB/sec execute 7 sec latency 22452.957 ms 1 869 0.00 MB/sec execute 8 sec latency 23453.231 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 970 0.02 MB/sec execute 9 sec latency 24157.353 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1422 0.44 MB/sec execute 10 sec latency 13.848 ms 1 1803 0.43 MB/sec execute 11 sec latency 18.546 ms 1 2400 0.53 MB/sec execute 12 sec latency 18.546 ms 1 2929 0.91 MB/sec execute 13 sec latency 22.720 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5284 1282404 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3400 1284288 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 59064 3541904 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 16364 3585348 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 75428 7127252 2% /mnt/lustre 1 3617 1.21 MB/sec execute 14 sec latency 22.555 ms 1 3890 1.21 MB/sec execute 15 sec latency 19.269 ms 1 4133 1.15 MB/sec execute 16 sec latency 20.776 ms 1 4538 1.13 MB/sec execute 17 sec latency 37.674 ms test_26 fail mds2 6 times Failing mds2 on oleg305-server Stopping /mnt/lustre-mds2 (opts:) on oleg305-server 1 5033 1.31 MB/sec execute 18 sec latency 15.707 ms 09:31:02 (1713533462) shut down 1 5699 1.27 MB/sec execute 19 sec latency 39.363 ms 1 5852 1.26 MB/sec execute 20 sec latency 780.583 ms 1 5852 1.20 MB/sec execute 21 sec latency 1780.732 ms 1 5852 1.15 MB/sec execute 22 sec latency 2780.868 ms 1 5852 1.10 MB/sec execute 23 sec latency 3781.001 ms 1 5852 1.05 MB/sec execute 24 sec latency 4781.130 ms 1 5852 1.01 MB/sec execute 25 sec latency 5781.241 ms 1 5852 0.97 MB/sec execute 26 sec latency 6781.327 ms 1 5852 0.93 MB/sec execute 27 sec latency 7781.440 ms 1 5852 0.90 MB/sec execute 28 sec latency 8781.541 ms Failover mds2 to oleg305-server mount facets: mds2 1 5852 0.87 MB/sec execute 29 sec latency 9781.642 ms 1 5852 0.84 MB/sec execute 30 sec latency 10781.734 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 5852 0.81 MB/sec execute 31 sec latency 11781.860 ms 1 5852 0.79 MB/sec execute 32 sec latency 12782.077 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0001 09:31:16 (1713533476) targets are mounted 09:31:16 (1713533476) facet_failover done 1 5852 0.76 MB/sec execute 33 sec latency 13782.336 ms 1 5852 0.74 MB/sec execute 34 sec latency 14782.500 ms 1 5852 0.72 MB/sec execute 35 sec latency 15782.655 ms 1 5852 0.70 MB/sec execute 36 sec latency 16782.840 ms 1 5889 0.68 MB/sec execute 37 sec latency 17709.652 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6168 0.70 MB/sec execute 38 sec latency 23.163 ms 1 6752 0.81 MB/sec execute 39 sec latency 19.677 ms 1 7190 0.90 MB/sec execute 40 sec latency 19.603 ms 1 7387 0.91 MB/sec execute 41 sec latency 25.797 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3068 1284620 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2504 1285184 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 49716 3553924 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 10872 3595212 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 60588 7149136 1% /mnt/lustre 1 7560 0.89 MB/sec execute 42 sec latency 57.419 ms 1 7902 0.88 MB/sec execute 43 sec latency 210.149 ms 1 8322 0.90 MB/sec execute 44 sec latency 363.465 ms 1 9018 0.96 MB/sec execute 45 sec latency 15.004 ms striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar test_26 fail mds1 7 times Failing mds1 on oleg305-server 1 9539 0.97 MB/sec execute 46 sec latency 40.973 ms Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 09:31:30 (1713533490) shut down 1 10096 1.07 MB/sec execute 47 sec latency 19.675 ms 1 10648 1.15 MB/sec execute 48 sec latency 19.830 ms 1 10929 1.15 MB/sec execute 49 sec latency 20.901 ms 1 11096 1.13 MB/sec execute 50 sec latency 21.084 ms 1 11174 1.11 MB/sec execute 51 sec latency 590.071 ms 1 11174 1.09 MB/sec execute 52 sec latency 1590.347 ms 1 11174 1.07 MB/sec execute 53 sec latency 2590.552 ms 1 11174 1.05 MB/sec execute 54 sec latency 3590.843 ms 1 11174 1.03 MB/sec execute 55 sec latency 4591.072 ms 1 11174 1.01 MB/sec execute 56 sec latency 5591.376 ms Failover mds1 to oleg305-server 1 11174 0.99 MB/sec execute 57 sec latency 6591.559 ms mount facets: mds1 1 11174 0.98 MB/sec execute 58 sec latency 7591.692 ms 1 11174 0.96 MB/sec execute 59 sec latency 8591.825 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11174 0.94 MB/sec execute 60 sec latency 9591.968 ms 1 11174 0.93 MB/sec execute 61 sec latency 10592.131 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 09:31:45 (1713533505) targets are mounted 09:31:45 (1713533505) facet_failover done 1 11174 0.91 MB/sec execute 62 sec latency 11592.341 ms 1 11174 0.90 MB/sec execute 63 sec latency 12592.581 ms 1 11174 0.89 MB/sec execute 64 sec latency 13592.902 ms 1 11174 0.87 MB/sec execute 65 sec latency 14593.122 ms 1 11174 0.86 MB/sec execute 66 sec latency 15593.377 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11461 0.85 MB/sec execute 67 sec latency 15797.337 ms 1 11824 0.86 MB/sec execute 68 sec latency 27.349 ms 1 12168 0.90 MB/sec execute 69 sec latency 21.587 ms 1 12546 0.89 MB/sec execute 70 sec latency 24.736 ms 1 13021 0.90 MB/sec execute 71 sec latency 20.792 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Apr 19 09:31:55 EDT 2024 with return code 0 10916 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 10916 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished