== replay-dual test 26: dbench and tar with mds failover ========================================================== 17:30:58 (1713389458) Starting client oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre Started clients oleg217-client.virtnet: 192.168.202.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 15825 Started dbench loop with 15826 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Apr 17 17:31:00 EDT 2024 waiting for dbench pid 15851 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 174 7.06 MB/sec warmup 1 sec latency 33.820 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3726336 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3724288 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 32768 7450624 1% /mnt/lustre 1 323 5.87 MB/sec warmup 2 sec latency 29.700 ms 1 485 5.65 MB/sec warmup 3 sec latency 28.305 ms 1 699 5.30 MB/sec warmup 4 sec latency 80.362 ms test_26 fail mds1 1 times Failing mds1 on oleg217-server 1 857 4.31 MB/sec warmup 5 sec latency 67.069 ms Stopping /mnt/lustre-mds1 (opts:) on oleg217-server 1 923 3.63 MB/sec warmup 6 sec latency 487.450 ms 1 923 3.11 MB/sec warmup 7 sec latency 1487.746 ms 1 923 2.72 MB/sec warmup 8 sec latency 2488.020 ms reboot facets: mds1 1 923 2.42 MB/sec warmup 9 sec latency 3488.227 ms 1 923 2.18 MB/sec warmup 10 sec latency 4488.455 ms 1 923 1.98 MB/sec warmup 11 sec latency 5488.682 ms 1 923 1.81 MB/sec warmup 12 sec latency 6488.934 ms 1 923 1.67 MB/sec warmup 13 sec latency 7489.175 ms 1 923 1.56 MB/sec warmup 14 sec latency 8489.415 ms 1 923 1.45 MB/sec warmup 15 sec latency 9489.659 ms 1 923 1.36 MB/sec warmup 16 sec latency 10489.875 ms 1 923 1.28 MB/sec warmup 17 sec latency 11490.112 ms 1 923 1.21 MB/sec warmup 18 sec latency 12490.329 ms Failover mds1 to oleg217-server mount facets: mds1 1 923 1.15 MB/sec warmup 19 sec latency 13490.517 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 923 0.00 MB/sec execute 1 sec latency 15490.844 ms 1 923 0.00 MB/sec execute 2 sec latency 16491.102 ms 1 923 0.00 MB/sec execute 3 sec latency 17491.354 ms oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all 8 1 923 0.00 MB/sec execute 4 sec latency 18491.595 ms pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 1 923 0.00 MB/sec execute 5 sec latency 19491.801 ms 1 923 0.00 MB/sec execute 6 sec latency 20492.098 ms 1 923 0.00 MB/sec execute 7 sec latency 21492.399 ms 1 923 0.00 MB/sec execute 8 sec latency 22492.586 ms 1 923 0.00 MB/sec execute 9 sec latency 23492.749 ms 1 923 0.00 MB/sec execute 10 sec latency 24492.918 ms 1 923 0.00 MB/sec execute 11 sec latency 25493.070 ms 1 923 0.00 MB/sec execute 12 sec latency 26493.268 ms 1 923 0.00 MB/sec execute 13 sec latency 27493.434 ms 1 923 0.00 MB/sec execute 14 sec latency 28493.591 ms 1 923 0.00 MB/sec execute 15 sec latency 29493.771 ms 1 923 0.00 MB/sec execute 16 sec latency 30493.922 ms 1 1011 0.03 MB/sec execute 17 sec latency 30906.650 ms oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1390 0.23 MB/sec execute 18 sec latency 42.726 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1651 0.23 MB/sec execute 19 sec latency 59.161 ms 1 1975 0.23 MB/sec execute 20 sec latency 43.864 ms 1 2337 0.29 MB/sec execute 21 sec latency 14.993 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4608 2204032 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3613696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3675136 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7288832 1% /mnt/lustre 1 2604 0.33 MB/sec execute 22 sec latency 44.843 ms 1 3255 0.57 MB/sec execute 23 sec latency 35.784 ms 1 3644 0.70 MB/sec execute 24 sec latency 28.720 ms test_26 fail mds1 2 times 1 3872 0.72 MB/sec execute 25 sec latency 23.283 ms Failing mds1 on oleg217-server Stopping /mnt/lustre-mds1 (opts:) on oleg217-server 1 4021 0.70 MB/sec execute 26 sec latency 155.301 ms reboot facets: mds1 1 4021 0.68 MB/sec execute 27 sec latency 1155.552 ms 1 4021 0.65 MB/sec execute 28 sec latency 2155.856 ms 1 4021 0.63 MB/sec execute 29 sec latency 3156.093 ms 1 4021 0.61 MB/sec execute 30 sec latency 4156.324 ms 1 4021 0.59 MB/sec execute 31 sec latency 5156.522 ms 1 4021 0.57 MB/sec execute 32 sec latency 6156.807 ms 1 4021 0.55 MB/sec execute 33 sec latency 7157.008 ms 1 4021 0.54 MB/sec execute 34 sec latency 8157.222 ms 1 4021 0.52 MB/sec execute 35 sec latency 9157.460 ms 1 4021 0.51 MB/sec execute 36 sec latency 10157.755 ms Failover mds1 to oleg217-server 1 4021 0.49 MB/sec execute 37 sec latency 11157.962 ms mount facets: mds1 1 4021 0.48 MB/sec execute 38 sec latency 12158.138 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4021 0.47 MB/sec execute 39 sec latency 13158.330 ms 1 4021 0.46 MB/sec execute 40 sec latency 14158.600 ms 1 4021 0.45 MB/sec execute 41 sec latency 15158.841 ms oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all 8 1 4021 0.44 MB/sec execute 42 sec latency 16159.040 ms pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4021 0.43 MB/sec execute 43 sec latency 17159.174 ms 1 4021 0.42 MB/sec execute 44 sec latency 18159.354 ms 1 4021 0.41 MB/sec execute 45 sec latency 19159.530 ms 1 4021 0.40 MB/sec execute 46 sec latency 20159.699 ms 1 4021 0.39 MB/sec execute 47 sec latency 21159.821 ms 1 4021 0.38 MB/sec execute 48 sec latency 22159.913 ms 1 4021 0.37 MB/sec execute 49 sec latency 23160.057 ms 1 4021 0.37 MB/sec execute 50 sec latency 24160.212 ms 1 4021 0.36 MB/sec execute 51 sec latency 25160.364 ms 1 4021 0.35 MB/sec execute 52 sec latency 26160.488 ms 1 4021 0.35 MB/sec execute 53 sec latency 27160.666 ms 1 4021 0.34 MB/sec execute 54 sec latency 28160.808 ms 1 4021 0.33 MB/sec execute 55 sec latency 29160.980 ms 1 4021 0.33 MB/sec execute 56 sec latency 30161.129 ms 1 4021 0.32 MB/sec execute 57 sec latency 31161.266 ms 1 4054 0.32 MB/sec execute 58 sec latency 31981.011 ms oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4297 0.32 MB/sec execute 59 sec latency 59.474 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4544 0.32 MB/sec execute 60 sec latency 47.837 ms 1 4899 0.36 MB/sec execute 61 sec latency 49.718 ms 1 5174 0.38 MB/sec execute 62 sec latency 49.266 ms 1 5406 0.38 MB/sec execute 63 sec latency 54.933 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 6144 2202496 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3653632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3613696 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 65536 7267328 1% /mnt/lustre 1 5765 0.38 MB/sec execute 64 sec latency 74.242 ms 1 5984 0.39 MB/sec execute 65 sec latency 59.948 ms 1 6303 0.44 MB/sec execute 66 sec latency 63.670 ms test_26 fail mds1 3 times Failing mds1 on oleg217-server 1 6566 0.46 MB/sec execute 67 sec latency 57.236 ms Stopping /mnt/lustre-mds1 (opts:) on oleg217-server 1 6767 0.47 MB/sec execute 68 sec latency 606.308 ms reboot facets: mds1 1 6767 0.46 MB/sec execute 69 sec latency 1606.495 ms 1 6767 0.45 MB/sec execute 70 sec latency 2606.693 ms 1 6767 0.45 MB/sec execute 71 sec latency 3606.986 ms 1 6767 0.44 MB/sec execute 72 sec latency 4607.177 ms 1 6767 0.43 MB/sec execute 73 sec latency 5607.373 ms 1 6767 0.43 MB/sec execute 74 sec latency 6607.609 ms 1 6767 0.42 MB/sec execute 75 sec latency 7607.929 ms 1 6767 0.42 MB/sec execute 76 sec latency 8608.203 ms 1 6767 0.41 MB/sec execute 77 sec latency 9608.420 ms 1 6767 0.41 MB/sec execute 78 sec latency 10608.632 ms Failover mds1 to oleg217-server mount facets: mds1 1 6767 0.40 MB/sec execute 79 sec latency 11608.842 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6767 0.40 MB/sec execute 80 sec latency 12609.027 ms 1 6767 0.39 MB/sec execute 81 sec latency 13609.215 ms 1 6767 0.39 MB/sec execute 82 sec latency 14609.461 ms 1 6767 0.38 MB/sec execute 83 sec latency 15609.664 ms 1 6767 0.38 MB/sec execute 84 sec latency 16609.859 ms 1 6767 0.37 MB/sec execute 85 sec latency 17610.014 ms oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 1 6767 0.37 MB/sec execute 86 sec latency 18610.222 ms 1 6767 0.36 MB/sec execute 87 sec latency 19610.357 ms 1 6767 0.36 MB/sec execute 88 sec latency 20610.464 ms 1 6767 0.36 MB/sec execute 89 sec latency 21610.606 ms 1 6767 0.35 MB/sec execute 90 sec latency 22610.808 ms 1 6767 0.35 MB/sec execute 91 sec latency 23610.934 ms 1 6767 0.34 MB/sec execute 92 sec latency 24611.017 ms 1 6767 0.34 MB/sec execute 93 sec latency 25611.127 ms 1 6767 0.34 MB/sec execute 94 sec latency 26611.239 ms 1 6767 0.33 MB/sec execute 95 sec latency 27611.342 ms 1 6767 0.33 MB/sec execute 96 sec latency 28611.521 ms 1 6767 0.33 MB/sec execute 97 sec latency 29611.661 ms 1 6767 0.32 MB/sec execute 98 sec latency 30611.791 ms 1 6767 0.32 MB/sec execute 99 sec latency 31611.886 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 974 41.342 30906.632 Close 748 1.916 4.623 Rename 45 13.908 22.469 Unlink 166 197.104 31980.989 Qpathinfo 958 36.447 32299.868 Qfileinfo 178 0.410 1.873 Qfsinfo 159 0.438 2.739 Sfileinfo 85 6.590 13.531 Find 378 1.045 12.220 WriteX 522 2.075 10.661 ReadX 1697 0.158 42.720 LockX 4 1.903 2.558 UnlockX 4 1.971 2.661 Flush 64 31.796 74.235 Throughput 0.31947 MB/sec 1 clients 1 procs max_latency=31981.011 ms stopping dbench on /mnt/lustre at Wed Apr 17 17:33:01 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Apr 17 17:33:02 EDT 2024 waiting for dbench pid 19222 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 239 9.14 MB/sec warmup 1 sec latency 27.281 ms 1 464 8.08 MB/sec warmup 2 sec latency 28.015 ms 1 728 7.10 MB/sec warmup 3 sec latency 92.579 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 7552 2201088 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 44032 3649536 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3690496 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7340032 1% /mnt/lustre 1 952 5.45 MB/sec warmup 4 sec latency 57.933 ms 1 1231 4.61 MB/sec warmup 5 sec latency 40.558 ms 1 1530 4.35 MB/sec warmup 6 sec latency 83.752 ms test_26 fail mds1 4 times Failing mds1 on oleg217-server 1 1745 3.75 MB/sec warmup 7 sec latency 70.587 ms Stopping /mnt/lustre-mds1 (opts:) on oleg217-server 1 1843 3.29 MB/sec warmup 8 sec latency 574.709 ms 1 1843 2.93 MB/sec warmup 9 sec latency 1574.915 ms 1 1843 2.64 MB/sec warmup 10 sec latency 2575.085 ms reboot facets: mds1 1 1843 2.40 MB/sec warmup 11 sec latency 3575.269 ms 1 1843 2.20 MB/sec warmup 12 sec latency 4575.486 ms 1 1843 2.03 MB/sec warmup 13 sec latency 5575.653 ms 1 1843 1.88 MB/sec warmup 14 sec latency 6575.838 ms 1 1843 1.76 MB/sec warmup 15 sec latency 7575.993 ms 1 1843 1.65 MB/sec warmup 16 sec latency 8576.141 ms 1 1843 1.55 MB/sec warmup 17 sec latency 9576.314 ms 1 1843 1.46 MB/sec warmup 18 sec latency 10576.552 ms 1 1843 1.39 MB/sec warmup 19 sec latency 11576.760 ms Failover mds1 to oleg217-server mount facets: mds1 1 1843 0.00 MB/sec execute 1 sec latency 13577.093 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1843 0.00 MB/sec execute 2 sec latency 14577.273 ms 1 1843 0.00 MB/sec execute 3 sec latency 15577.484 ms 1 1843 0.00 MB/sec execute 4 sec latency 16577.691 ms 1 1843 0.00 MB/sec execute 5 sec latency 17577.851 ms 1 1843 0.00 MB/sec execute 6 sec latency 18577.961 ms oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 1 1843 0.00 MB/sec execute 7 sec latency 19578.065 ms Started lustre-MDT0000 1 1843 0.00 MB/sec execute 8 sec latency 20578.168 ms 1 1843 0.00 MB/sec execute 9 sec latency 21578.324 ms 1 1843 0.00 MB/sec execute 10 sec latency 22578.471 ms 1 1843 0.00 MB/sec execute 11 sec latency 23578.591 ms 1 1843 0.00 MB/sec execute 12 sec latency 24578.699 ms 1 1843 0.00 MB/sec execute 13 sec latency 25578.833 ms 1 1843 0.00 MB/sec execute 14 sec latency 26578.981 ms 1 1843 0.00 MB/sec execute 15 sec latency 27579.146 ms 1 1843 0.00 MB/sec execute 16 sec latency 28579.330 ms 1 1843 0.00 MB/sec execute 17 sec latency 29579.519 ms 1 1843 0.00 MB/sec execute 18 sec latency 30579.657 ms 1 1843 0.00 MB/sec execute 19 sec latency 31579.816 ms 1 1843 0.00 MB/sec execute 20 sec latency 32579.956 ms 1 2168 0.02 MB/sec execute 21 sec latency 32679.867 ms oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2452 0.08 MB/sec execute 22 sec latency 64.242 ms 1 2934 0.31 MB/sec execute 23 sec latency 56.483 ms 1 3388 0.38 MB/sec execute 24 sec latency 44.992 ms 1 3785 0.54 MB/sec execute 25 sec latency 25.042 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 8960 2199680 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 51200 3692544 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3709952 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 71680 7402496 1% /mnt/lustre 1 3995 0.53 MB/sec execute 26 sec latency 26.048 ms 1 4270 0.52 MB/sec execute 27 sec latency 62.963 ms 1 4472 0.52 MB/sec execute 28 sec latency 58.711 ms test_26 fail mds1 5 times Failing mds1 on oleg217-server 1 4739 0.54 MB/sec execute 29 sec latency 50.639 ms Stopping /mnt/lustre-mds1 (opts:) on oleg217-server 1 4829 0.53 MB/sec execute 30 sec latency 656.962 ms reboot facets: mds1 1 4829 0.51 MB/sec execute 31 sec latency 1657.250 ms 1 4829 0.50 MB/sec execute 32 sec latency 2657.600 ms 1 4829 0.48 MB/sec execute 33 sec latency 3657.853 ms 1 4829 0.47 MB/sec execute 34 sec latency 4658.124 ms 1 4829 0.45 MB/sec execute 35 sec latency 5658.402 ms 1 4829 0.44 MB/sec execute 36 sec latency 6658.708 ms 1 4829 0.43 MB/sec execute 37 sec latency 7658.995 ms 1 4829 0.42 MB/sec execute 38 sec latency 8659.293 ms 1 4829 0.41 MB/sec execute 39 sec latency 9659.589 ms 1 4829 0.40 MB/sec execute 40 sec latency 10659.847 ms Failover mds1 to oleg217-server mount facets: mds1 1 4829 0.39 MB/sec execute 41 sec latency 11660.086 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4829 0.38 MB/sec execute 42 sec latency 12660.357 ms 1 4829 0.37 MB/sec execute 43 sec latency 13660.575 ms 1 4829 0.36 MB/sec execute 44 sec latency 14660.811 ms 1 4829 0.35 MB/sec execute 45 sec latency 15661.039 ms oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4829 0.35 MB/sec execute 46 sec latency 16661.271 ms 1 4829 0.34 MB/sec execute 47 sec latency 17661.407 ms 1 5078 0.39 MB/sec execute 48 sec latency 17674.353 ms oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5340 0.39 MB/sec execute 49 sec latency 28.459 ms 1 5561 0.39 MB/sec execute 50 sec latency 52.330 ms 1 5798 0.38 MB/sec execute 51 sec latency 26.088 ms 1 5996 0.40 MB/sec execute 52 sec latency 72.784 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10496 2198144 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 55296 3443712 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 28672 3474432 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 83968 6918144 2% /mnt/lustre 1 6317 0.47 MB/sec execute 53 sec latency 73.384 ms 1 6752 0.50 MB/sec execute 54 sec latency 65.340 ms 1 7124 0.57 MB/sec execute 55 sec latency 29.320 ms test_26 fail mds1 6 times 1 7368 0.58 MB/sec execute 56 sec latency 25.043 ms Failing mds1 on oleg217-server Stopping /mnt/lustre-mds1 (opts:) on oleg217-server 1 7566 0.58 MB/sec execute 57 sec latency 59.003 ms reboot facets: mds1 1 7566 0.57 MB/sec execute 58 sec latency 1059.126 ms 1 7566 0.56 MB/sec execute 59 sec latency 2059.246 ms 1 7566 0.55 MB/sec execute 60 sec latency 3059.391 ms 1 7566 0.54 MB/sec execute 61 sec latency 4059.525 ms 1 7566 0.53 MB/sec execute 62 sec latency 5059.654 ms 1 7566 0.52 MB/sec execute 63 sec latency 6059.805 ms 1 7566 0.51 MB/sec execute 64 sec latency 7059.927 ms 1 7566 0.50 MB/sec execute 65 sec latency 8060.131 ms 1 7566 0.50 MB/sec execute 66 sec latency 9060.331 ms 1 7566 0.49 MB/sec execute 67 sec latency 10060.475 ms Failover mds1 to oleg217-server 1 7566 0.48 MB/sec execute 68 sec latency 11060.600 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7566 0.48 MB/sec execute 69 sec latency 12060.702 ms 1 7566 0.47 MB/sec execute 70 sec latency 13060.805 ms 1 7566 0.46 MB/sec execute 71 sec latency 14060.967 ms 1 7566 0.46 MB/sec execute 72 sec latency 15061.144 ms 1 7566 0.45 MB/sec execute 73 sec latency 16061.256 ms oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 1 7566 0.44 MB/sec execute 74 sec latency 17061.416 ms 1 7566 0.44 MB/sec execute 75 sec latency 18061.533 ms 1 7566 0.43 MB/sec execute 76 sec latency 19061.627 ms 1 7566 0.43 MB/sec execute 77 sec latency 20061.858 ms 1 7566 0.42 MB/sec execute 78 sec latency 21062.056 ms 1 7566 0.42 MB/sec execute 79 sec latency 22062.170 ms 1 7566 0.41 MB/sec execute 80 sec latency 23062.275 ms 1 7566 0.41 MB/sec execute 81 sec latency 24062.372 ms 1 7566 0.40 MB/sec execute 82 sec latency 25062.489 ms 1 7566 0.40 MB/sec execute 83 sec latency 26062.608 ms 1 7566 0.39 MB/sec execute 84 sec latency 27062.795 ms 1 7566 0.39 MB/sec execute 85 sec latency 28063.015 ms 1 7566 0.38 MB/sec execute 86 sec latency 29063.195 ms 1 7566 0.38 MB/sec execute 87 sec latency 30063.339 ms 1 7768 0.37 MB/sec execute 88 sec latency 30101.474 ms oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7922 0.37 MB/sec execute 89 sec latency 103.291 ms 1 8091 0.38 MB/sec execute 90 sec latency 65.774 ms kill: sending signal to 15838 failed: No such process 1 8309 0.38 MB/sec execute 91 sec latency 57.139 ms 1 8590 0.41 MB/sec execute 92 sec latency 26.865 ms 1 8742 0.41 MB/sec execute 93 sec latency 92.950 ms 1 8938 0.41 MB/sec execute 94 sec latency 29.823 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Apr 17 17:34:56 EDT 2024 with return code 0 19222 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 19222 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished