== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 04:21:41 (1713428501) Starting client oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg457-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=21605 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1285564 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3592224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26128 3556172 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27712 7148396 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:21:49 (1713428509) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:22:05 (1713428525) targets are mounted 04:22:05 (1713428525) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2092 1285596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3592968 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26392 3569804 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28204 7162772 1% /mnt/lustre test_70b fail mds2 2 times oleg457-client.virtnet: looking for dbench program oleg457-client.virtnet: /usr/bin/dbench oleg457-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg457-client.virtnet oleg457-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg457-client.virtnet' oleg457-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg457-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg457-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg457-client.virtnet at Thu Apr 18 04:21:42 EDT 2024 oleg457-client.virtnet: waiting for dbench pid 21629 oleg457-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg457-client.virtnet: oleg457-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg457-client.virtnet: failed to create barrier semaphore oleg457-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg457-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg457-client.virtnet: releasing clients oleg457-client.virtnet: 1 480 16.69 MB/sec warmup 1 sec latency 15.306 ms oleg457-client.virtnet: 1 1074 11.46 MB/sec warmup 2 sec latency 12.161 ms oleg457-client.virtnet: 1 1706 8.74 MB/sec warmup 3 sec latency 11.690 ms oleg457-client.virtnet: 1 2356 6.97 MB/sec warmup 4 sec latency 15.360 ms oleg457-client.virtnet: 1 2918 6.69 MB/sec warmup 5 sec latency 15.430 ms oleg457-client.virtnet: 1 3592 6.41 MB/sec warmup 6 sec latency 14.530 ms oleg457-client.virtnet: 1 3931 5.72 MB/sec warmup 7 sec latency 18.715 ms oleg457-client.virtnet: 1 4083 5.01 MB/sec warmup 8 sec latency 349.420 ms oleg457-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 1349.680 ms oleg457-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 2349.957 ms oleg457-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 3350.185 ms oleg457-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 4350.505 ms oleg457-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 5350.774 ms oleg457-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 6351.022 ms oleg457-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 7351.379 ms oleg457-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 8351.589 ms oleg457-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 9351.785 ms oleg457-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 10351.930 ms oleg457-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 11352.113 ms oleg457-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 12352.374 ms oleg457-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 13352.559 ms oleg457-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 14352.753 ms oleg457-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 15353.053 ms oleg457-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 16353.416 ms oleg457-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 17353.761 ms oleg457-client.virtnet: 1 4314 1.56 MB/sec warmup 26 sec latency 17789.666 ms oleg457-client.virtnet: 1 4667 1.56 MB/sec warmup 27 sec latency 19.118 ms oleg457-client.virtnet: 1 5143 1.62 MB/sec warmup 28 sec latency 15.402 ms oleg457-client.virtnet: 1 5759 1.58 MB/sec warmup 29 sec latency 11.002 ms oleg457-client.virtnet: 1 6319 1.70 MB/sec warmup 30 sec latency 15.802 ms oleg457-client.virtnet: 1 7134 1.86 MB/sec warmup 31 sec latency 14.682 ms oleg457-client.virtnet: 1 7503 1.85 MB/sec warmup 32 sec latency 15.870 ms oleg457-client.virtnet: 1 7824 1.80 MB/sec warmup 33 sec latency 19.0Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:22:17 (1713428537) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:22:31 (1713428551) targets are mounted 04:22:31 (1713428551) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13136 3592964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36380 3570020 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49516 7162984 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:22:44 (1713428564) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:23:00 (1713428580) targets are mounted 04:23:00 (1713428580) facet_failover done 99 ms oleg457-client.virtnet: 1 8258 1.80 MB/sec warmup 34 sec latency 169.316 ms oleg457-client.virtnet: 1 8258 1.75 MB/sec warmup 35 sec latency 1169.501 ms oleg457-client.virtnet: 1 8258 1.70 MB/sec warmup 36 sec latency 2169.738 ms oleg457-client.virtnet: 1 8258 1.65 MB/sec warmup 37 sec latency 3170.067 ms oleg457-client.virtnet: 1 8258 1.61 MB/sec warmup 38 sec latency 4170.349 ms oleg457-client.virtnet: 1 8258 1.57 MB/sec warmup 39 sec latency 5170.671 ms oleg457-client.virtnet: 1 8258 1.53 MB/sec warmup 40 sec latency 6170.936 ms oleg457-client.virtnet: 1 8258 1.49 MB/sec warmup 41 sec latency 7171.182 ms oleg457-client.virtnet: 1 8258 1.46 MB/sec warmup 42 sec latency 8171.445 ms oleg457-client.virtnet: 1 8258 1.42 MB/sec warmup 43 sec latency 9171.672 ms oleg457-client.virtnet: 1 8258 1.39 MB/sec warmup 44 sec latency 10171.896 ms oleg457-client.virtnet: 1 8258 1.36 MB/sec warmup 45 sec latency 11172.155 ms oleg457-client.virtnet: 1 8258 1.33 MB/sec warmup 46 sec latency 12172.411 ms oleg457-client.virtnet: 1 8258 1.30 MB/sec warmup 47 sec latency 13172.615 ms oleg457-client.virtnet: 1 8258 1.28 MB/sec warmup 48 sec latency 14172.784 ms oleg457-client.virtnet: 1 8258 1.25 MB/sec warmup 49 sec latency 15172.982 ms oleg457-client.virtnet: 1 8258 1.22 MB/sec warmup 50 sec latency 16173.236 ms oleg457-client.virtnet: 1 8258 1.20 MB/sec warmup 51 sec latency 17173.657 ms oleg457-client.virtnet: 1 8258 1.18 MB/sec warmup 52 sec latency 18173.913 ms oleg457-client.virtnet: 1 8258 1.16 MB/sec warmup 53 sec latency 19174.144 ms oleg457-client.virtnet: 1 8647 1.19 MB/sec warmup 54 sec latency 19426.326 ms oleg457-client.virtnet: 1 9107 1.18 MB/sec warmup 55 sec latency 17.065 ms oleg457-client.virtnet: 1 9657 1.20 MB/sec warmup 56 sec latency 12.320 ms oleg457-client.virtnet: 1 10440 1.29 MB/sec warmup 57 sec latency 15.578 ms oleg457-client.virtnet: 1 10974 1.34 MB/sec warmup 58 sec latency 11.706 ms oleg457-client.virtnet: 1 11357 1.33 MB/sec warmup 59 sec latency 19.289 ms oleg457-client.virtnet: 1 12287 4.26 MB/sec execute 1 sec latency 16.173 ms oleg457-client.virtnet: 1 12884 2.44 MB/sec execute 2 sec latency 9.538 ms oleg457-client.virtnet: 1 13261 2.37 MB/sec execute 3 sec latency 15.962 ms oleg457-client.virtnet: 1 14268 4.17 MB/sec execute 4 sec latency 10.762 ms oleg457-client.virtnet: 1 14554 3.63 MB/sec execute 5 sec latency 16.000 ms oleg457-client.virtnet: 1 15027 3.12 MB/sec execute 6 sec latency 13.952 ms oleg457-client.virtnet: 1 15404 2.89 MB/sec execute 7 sec latency 16.573 ms oleg457-client.virtnet: 1 15937 2.94 MB/sec execute 8 sec latency 15.692 ms oleg457-client.virtnet: 1 16563 2.79 MB/sec execute 9 sec latency 11.247 ms oleg457-client.virtnet: 1 17084 2.99 MB/sec execute 10 sec latency 15.154 ms oleg457-client.virtnet: 1 17715 3.23 MB/sec execute 11 sec latency 16.522 ms oleg457-client.virtnet: 1 18020 3.08 MB/sec execute 12 sec latency 15.894 ms oleg457-client.virtnet: 1 18259 2.87 MB/sec execute 13 sec latency 15.673 ms oleg457-client.virtnet: 1 18621 2.71 MB/sec execute 14 sec latency 23.024 ms oleg457-client.virtnet: 1 18964 2.63 MB/sec execute 15 sec latency 15.496 ms oleg457-client.virtnet: 1 19374 2.66 MB/sec execute 16 sec latency 15.696 ms oleg457-client.virtnet: 1 19751 2.52 MB/sec execute 17 sec latency 16.470 ms oleg457-client.virtnet: 1 20152 2.46 MB/sec execute 18 sec latency 13.988 ms oleg457-client.virtnet: 1 20657 2.56 MB/sec execuoleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2048 1285640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13068 3592960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36556 3569108 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7162068 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:23:12 (1713428592) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:23:28 (1713428608) targets are mounted 04:23:28 (1713428608) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13068 3591884 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36532 3567656 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49600 7159540 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:23:41 (1713428621) shut down te 19 sec latency 15.811 ms oleg457-client.virtnet: 1 21284 2.74 MB/sec execute 20 sec latency 15.615 ms oleg457-client.virtnet: 1 21601 2.67 MB/sec execute 21 sec latency 16.303 ms oleg457-client.virtnet: 1 21861 2.57 MB/sec execute 22 sec latency 14.167 ms oleg457-client.virtnet: 1 22208 2.49 MB/sec execute 23 sec latency 21.481 ms oleg457-client.virtnet: 1 22562 2.44 MB/sec execute 24 sec latency 15.735 ms oleg457-client.virtnet: 1 23053 2.47 MB/sec execute 25 sec latency 16.744 ms oleg457-client.virtnet: 1 23425 2.39 MB/sec execute 26 sec latency 12.996 ms oleg457-client.virtnet: 1 23783 2.36 MB/sec execute 27 sec latency 16.345 ms oleg457-client.virtnet: 1 24335 2.47 MB/sec execute 28 sec latency 16.255 ms oleg457-client.virtnet: 1 24960 2.56 MB/sec execute 29 sec latency 17.374 ms oleg457-client.virtnet: 1 25039 2.48 MB/sec execute 30 sec latency 721.537 ms oleg457-client.virtnet: 1 25039 2.40 MB/sec execute 31 sec latency 1721.817 ms oleg457-client.virtnet: 1 25039 2.32 MB/sec execute 32 sec latency 2722.058 ms oleg457-client.virtnet: 1 25039 2.25 MB/sec execute 33 sec latency 3722.282 ms oleg457-client.virtnet: 1 25039 2.19 MB/sec execute 34 sec latency 4722.552 ms oleg457-client.virtnet: 1 25039 2.12 MB/sec execute 35 sec latency 5722.749 ms oleg457-client.virtnet: 1 25039 2.06 MB/sec execute 36 sec latency 6723.050 ms oleg457-client.virtnet: 1 25039 2.01 MB/sec execute 37 sec latency 7723.264 ms oleg457-client.virtnet: 1 25039 1.96 MB/sec execute 38 sec latency 8723.540 ms oleg457-client.virtnet: 1 25039 1.90 MB/sec execute 39 sec latency 9723.775 ms oleg457-client.virtnet: 1 25039 1.86 MB/sec execute 40 sec latency 10724.027 ms oleg457-client.virtnet: 1 25039 1.81 MB/sec execute 41 sec latency 11724.291 ms oleg457-client.virtnet: 1 25039 1.77 MB/sec execute 42 sec latency 12724.497 ms oleg457-client.virtnet: 1 25039 1.73 MB/sec execute 43 sec latency 13724.689 ms oleg457-client.virtnet: 1 25039 1.69 MB/sec execute 44 sec latency 14724.900 ms oleg457-client.virtnet: 1 25039 1.65 MB/sec execute 45 sec latency 15725.132 ms oleg457-client.virtnet: 1 25039 1.62 MB/sec execute 46 sec latency 16725.370 ms oleg457-client.virtnet: 1 25039 1.58 MB/sec execute 47 sec latency 17725.599 ms oleg457-client.virtnet: 1 25039 1.55 MB/sec execute 48 sec latency 18725.852 ms oleg457-client.virtnet: 1 25039 1.52 MB/sec execute 49 sec latency 19726.057 ms oleg457-client.virtnet: 1 25274 1.51 MB/sec execute 50 sec latency 19825.998 ms oleg457-client.virtnet: 1 25585 1.49 MB/sec execute 51 sec latency 17.081 ms oleg457-client.virtnet: 1 26058 1.49 MB/sec execute 52 sec latency 14.920 ms oleg457-client.virtnet: 1 26449 1.52 MB/sec execute 53 sec latency 18.327 ms oleg457-client.virtnet: 1 26749 1.50 MB/sec execute 54 sec latency 16.871 ms oleg457-client.virtnet: 1 27177 1.50 MB/sec execute 55 sec latency 11.635 ms oleg457-client.virtnet: 1 27742 1.55 MB/sec execute 56 sec latency 16.508 ms oleg457-client.virtnet: 1 28385 1.63 MB/sec execute 57 sec latency 15.123 ms oleg457-client.virtnet: 1 28718 1.63 MB/sec execute 58 sec latency 15.523 ms oleg457-client.virtnet: 1 29005 1.61 MB/sec execute 59 sec latency 15.404 ms oleg457-client.virtnet: 1 29362 1.59 MB/sec execute 60 sec latency 16.124 ms oleg457-client.virtnet: 1 29827 1.64 MB/sec execute 61 sec latency 15.974 ms oleg457-client.virtnet: 1 30146 1.61 MB/sec execute 62 sec latency 17.982 ms oleg457-client.virtnet: 1 30518 1.59 MB/sec execute 63 sec latency 13.755 ms oleg457-clFailover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:23:57 (1713428637) targets are mounted 04:23:57 (1713428637) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2044 1285644 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13228 3592000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36424 3569896 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49652 7161896 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:24:10 (1713428650) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:24:25 (1713428665) targets are mounted 04:24:25 (1713428665) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec ient.virtnet: 1 30878 1.59 MB/sec execute 64 sec latency 15.067 ms oleg457-client.virtnet: 1 31440 1.65 MB/sec execute 65 sec latency 15.982 ms oleg457-client.virtnet: 1 32297 1.72 MB/sec execute 66 sec latency 15.317 ms oleg457-client.virtnet: 1 32584 1.70 MB/sec execute 67 sec latency 16.342 ms oleg457-client.virtnet: 1 32888 1.69 MB/sec execute 68 sec latency 23.730 ms oleg457-client.virtnet: 1 33234 1.68 MB/sec execute 69 sec latency 16.116 ms oleg457-client.virtnet: 1 33626 1.70 MB/sec execute 70 sec latency 15.960 ms oleg457-client.virtnet: 1 33976 1.68 MB/sec execute 71 sec latency 18.814 ms oleg457-client.virtnet: 1 34384 1.68 MB/sec execute 72 sec latency 15.173 ms oleg457-client.virtnet: 1 34955 1.73 MB/sec execute 73 sec latency 15.245 ms oleg457-client.virtnet: 1 35583 1.78 MB/sec execute 74 sec latency 14.705 ms oleg457-client.virtnet: 1 35864 1.77 MB/sec execute 75 sec latency 16.632 ms oleg457-client.virtnet: 1 36176 1.75 MB/sec execute 76 sec latency 15.955 ms oleg457-client.virtnet: 1 36454 1.74 MB/sec execute 77 sec latency 20.286 ms oleg457-client.virtnet: 1 36811 1.74 MB/sec execute 78 sec latency 16.234 ms oleg457-client.virtnet: 1 37200 1.75 MB/sec execute 79 sec latency 16.256 ms oleg457-client.virtnet: 1 37534 1.73 MB/sec execute 80 sec latency 16.626 ms oleg457-client.virtnet: 1 37926 1.73 MB/sec execute 81 sec latency 14.660 ms oleg457-client.virtnet: 1 38448 1.77 MB/sec execute 82 sec latency 16.171 ms oleg457-client.virtnet: 1 39173 1.82 MB/sec execute 83 sec latency 15.025 ms oleg457-client.virtnet: 1 39447 1.81 MB/sec execute 84 sec latency 15.706 ms oleg457-client.virtnet: 1 39764 1.79 MB/sec execute 85 sec latency 19.785 ms oleg457-client.virtnet: 1 40048 1.78 MB/sec execute 86 sec latency 16.430 ms oleg457-client.virtnet: 1 40229 1.77 MB/sec execute 87 sec latency 507.482 ms oleg457-client.virtnet: 1 40229 1.75 MB/sec execute 88 sec latency 1507.716 ms oleg457-client.virtnet: 1 40229 1.73 MB/sec execute 89 sec latency 2507.957 ms oleg457-client.virtnet: 1 40229 1.71 MB/sec execute 90 sec latency 3508.152 ms oleg457-client.virtnet: 1 40229 1.69 MB/sec execute 91 sec latency 4508.443 ms oleg457-client.virtnet: 1 40229 1.67 MB/sec execute 92 sec latency 5508.697 ms oleg457-client.virtnet: 1 40229 1.66 MB/sec execute 93 sec latency 6508.966 ms oleg457-client.virtnet: 1 40229 1.64 MB/sec execute 94 sec latency 7509.171 ms oleg457-client.virtnet: 1 40229 1.62 MB/sec execute 95 sec latency 8509.446 ms oleg457-client.virtnet: 1 40229 1.61 MB/sec execute 96 sec latency 9509.727 ms oleg457-client.virtnet: 1 40229 1.59 MB/sec execute 97 sec latency 10509.920 ms oleg457-client.virtnet: 1 40229 1.57 MB/sec execute 98 sec latency 11510.066 ms oleg457-client.virtnet: 1 40229 1.56 MB/sec execute 99 sec latency 12510.256 ms oleg457-client.virtnet: 1 40229 1.54 MB/sec execute 100 sec latency 13510.470 ms oleg457-client.virtnet: 1 40229 1.53 MB/sec execute 101 sec latency 14511.968 ms oleg457-client.virtnet: 1 40229 1.51 MB/sec execute 102 sec latency 15512.176 ms oleg457-client.virtnet: 1 40229 1.50 MB/sec execute 103 sec latency 16512.435 ms oleg457-client.virtnet: 1 40229 1.48 MB/sec execute 104 sec latency 17512.621 ms oleg457-client.virtnet: 1 40229 1.47 MB/sec execute 105 sec latency 18512.835 ms oleg457-client.virtnet: 1 40229 1.45 MB/sec execute 106 sec latency 19513.082 ms oleg457-client.virtnet: 1 40484 1.47 MB/sec execute 107 sec latency 19946.969 ms oleg457-client.virtnet: 1 40775 1.46 MBUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13244 3592792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36512 3569976 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49756 7162768 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:24:39 (1713428679) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:24:54 (1713428694) targets are mounted 04:24:54 (1713428694) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13116 3590816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36476 3569748 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49592 7160564 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:25:07 (1713428707) shut down /sec execute 108 sec latency 17.624 ms oleg457-client.virtnet: 1 41120 1.45 MB/sec execute 109 sec latency 12.155 ms oleg457-client.virtnet: 1 41485 1.45 MB/sec execute 110 sec latency 16.171 ms oleg457-client.virtnet: 1 42047 1.48 MB/sec execute 111 sec latency 16.397 ms oleg457-client.virtnet: 1 42662 1.52 MB/sec execute 112 sec latency 16.620 ms oleg457-client.virtnet: 1 42928 1.51 MB/sec execute 113 sec latency 17.603 ms oleg457-client.virtnet: 1 43203 1.50 MB/sec execute 114 sec latency 16.369 ms oleg457-client.virtnet: 1 43520 1.50 MB/sec execute 115 sec latency 18.385 ms oleg457-client.virtnet: 1 43855 1.49 MB/sec execute 116 sec latency 18.338 ms oleg457-client.virtnet: 1 44246 1.51 MB/sec execute 117 sec latency 15.712 ms oleg457-client.virtnet: 1 44603 1.50 MB/sec execute 118 sec latency 16.948 ms oleg457-client.virtnet: 1 44993 1.50 MB/sec execute 119 sec latency 16.404 ms oleg457-client.virtnet: 1 45513 1.53 MB/sec execute 120 sec latency 15.586 ms oleg457-client.virtnet: 1 46142 1.56 MB/sec execute 121 sec latency 16.191 ms oleg457-client.virtnet: 1 46527 1.56 MB/sec execute 122 sec latency 15.233 ms oleg457-client.virtnet: 1 46831 1.55 MB/sec execute 123 sec latency 22.856 ms oleg457-client.virtnet: 1 47126 1.54 MB/sec execute 124 sec latency 16.146 ms oleg457-client.virtnet: 1 47543 1.56 MB/sec execute 125 sec latency 18.098 ms oleg457-client.virtnet: 1 47843 1.55 MB/sec execute 126 sec latency 18.512 ms oleg457-client.virtnet: 1 48213 1.54 MB/sec execute 127 sec latency 10.823 ms oleg457-client.virtnet: 1 48601 1.54 MB/sec execute 128 sec latency 15.624 ms oleg457-client.virtnet: 1 49551 1.59 MB/sec execute 129 sec latency 13.912 ms oleg457-client.virtnet: 1 49977 1.61 MB/sec execute 130 sec latency 15.802 ms oleg457-client.virtnet: 1 50219 1.60 MB/sec execute 131 sec latency 15.899 ms oleg457-client.virtnet: 1 50571 1.59 MB/sec execute 132 sec latency 15.797 ms oleg457-client.virtnet: 1 50894 1.59 MB/sec execute 133 sec latency 15.976 ms oleg457-client.virtnet: 1 51375 1.60 MB/sec execute 134 sec latency 15.725 ms oleg457-client.virtnet: 1 51727 1.59 MB/sec execute 135 sec latency 14.216 ms oleg457-client.virtnet: 1 52126 1.59 MB/sec execute 136 sec latency 15.371 ms oleg457-client.virtnet: 1 52669 1.62 MB/sec execute 137 sec latency 15.961 ms oleg457-client.virtnet: 1 53270 1.64 MB/sec execute 138 sec latency 16.076 ms oleg457-client.virtnet: 1 53571 1.64 MB/sec execute 139 sec latency 15.316 ms oleg457-client.virtnet: 1 53839 1.63 MB/sec execute 140 sec latency 17.006 ms oleg457-client.virtnet: 1 54037 1.62 MB/sec execute 141 sec latency 369.500 ms oleg457-client.virtnet: 1 54286 1.62 MB/sec execute 142 sec latency 584.665 ms oleg457-client.virtnet: 1 54690 1.63 MB/sec execute 143 sec latency 15.543 ms oleg457-client.virtnet: 1 54967 1.62 MB/sec execute 144 sec latency 84.269 ms oleg457-client.virtnet: 1 54967 1.61 MB/sec execute 145 sec latency 1084.550 ms oleg457-client.virtnet: 1 54967 1.60 MB/sec execute 146 sec latency 2084.754 ms oleg457-client.virtnet: 1 54967 1.59 MB/sec execute 147 sec latency 3084.981 ms oleg457-client.virtnet: 1 54967 1.58 MB/sec execute 148 sec latency 4085.213 ms oleg457-client.virtnet: 1 54967 1.57 MB/sec execute 149 sec latency 5085.363 ms oleg457-client.virtnet: 1 54967 1.56 MB/sec execute 150 sec latency 6085.557 ms oleg457-client.virtnet: 1 54967 1.55 MB/sec execute 151 sec latency 7085.764 ms oleg457-client.virtnet: 1 54967 1.54 MB/sec execute 152 sec latency 8086.030 ms oleg457-client.virtnet: 1 Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:25:23 (1713428723) targets are mounted 04:25:23 (1713428723) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13112 3592552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36384 3567960 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7160512 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:25:36 (1713428736) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:25:51 (1713428751) targets are mounted 04:25:51 (1713428751) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593084 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36544 3569468 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7162552 1% /mnt/lustre 54967 1.53 MB/sec execute 153 sec latency 9086.291 ms oleg457-client.virtnet: 1 54967 1.52 MB/sec execute 154 sec latency 10086.523 ms oleg457-client.virtnet: 1 54967 1.51 MB/sec execute 155 sec latency 11086.717 ms oleg457-client.virtnet: 1 54967 1.50 MB/sec execute 156 sec latency 12086.883 ms oleg457-client.virtnet: 1 54967 1.49 MB/sec execute 157 sec latency 13087.143 ms oleg457-client.virtnet: 1 54967 1.48 MB/sec execute 158 sec latency 14087.382 ms oleg457-client.virtnet: 1 54967 1.47 MB/sec execute 159 sec latency 15087.640 ms oleg457-client.virtnet: 1 54967 1.46 MB/sec execute 160 sec latency 16087.850 ms oleg457-client.virtnet: 1 54967 1.45 MB/sec execute 161 sec latency 17088.125 ms oleg457-client.virtnet: 1 54967 1.44 MB/sec execute 162 sec latency 18088.403 ms oleg457-client.virtnet: 1 54967 1.43 MB/sec execute 163 sec latency 19088.822 ms oleg457-client.virtnet: 1 55044 1.43 MB/sec execute 164 sec latency 19837.150 ms oleg457-client.virtnet: 1 55436 1.42 MB/sec execute 165 sec latency 12.650 ms oleg457-client.virtnet: 1 55842 1.43 MB/sec execute 166 sec latency 16.819 ms oleg457-client.virtnet: 1 56587 1.45 MB/sec execute 167 sec latency 16.097 ms oleg457-client.virtnet: 1 56997 1.47 MB/sec execute 168 sec latency 15.214 ms oleg457-client.virtnet: 1 57243 1.47 MB/sec execute 169 sec latency 18.184 ms oleg457-client.virtnet: 1 57569 1.46 MB/sec execute 170 sec latency 19.133 ms oleg457-client.virtnet: 1 57888 1.46 MB/sec execute 171 sec latency 17.102 ms oleg457-client.virtnet: 1 58284 1.47 MB/sec execute 172 sec latency 15.974 ms oleg457-client.virtnet: 1 58607 1.46 MB/sec execute 173 sec latency 16.890 ms oleg457-client.virtnet: 1 58992 1.46 MB/sec execute 174 sec latency 11.252 ms oleg457-client.virtnet: 1 59376 1.46 MB/sec execute 175 sec latency 16.125 ms oleg457-client.virtnet: 1 60119 1.49 MB/sec execute 176 sec latency 15.499 ms oleg457-client.virtnet: 1 60547 1.51 MB/sec execute 177 sec latency 15.436 ms oleg457-client.virtnet: 1 60778 1.50 MB/sec execute 178 sec latency 15.683 ms oleg457-client.virtnet: 1 61102 1.49 MB/sec execute 179 sec latency 19.629 ms oleg457-client.virtnet: 1 61414 1.49 MB/sec execute 180 sec latency 16.017 ms oleg457-client.virtnet: 1 61822 1.50 MB/sec execute 181 sec latency 16.096 ms oleg457-client.virtnet: 1 62332 1.50 MB/sec execute 182 sec latency 13.091 ms oleg457-client.virtnet: 1 62748 1.50 MB/sec execute 183 sec latency 15.295 ms oleg457-client.virtnet: 1 63263 1.52 MB/sec execute 184 sec latency 16.405 ms oleg457-client.virtnet: 1 63890 1.54 MB/sec execute 185 sec latency 15.885 ms oleg457-client.virtnet: 1 64179 1.54 MB/sec execute 186 sec latency 16.463 ms oleg457-client.virtnet: 1 64548 1.53 MB/sec execute 187 sec latency 15.684 ms oleg457-client.virtnet: 1 65076 1.53 MB/sec execute 188 sec latency 13.108 ms oleg457-client.virtnet: 1 65473 1.54 MB/sec execute 189 sec latency 15.292 ms oleg457-client.virtnet: 1 65800 1.53 MB/sec execute 190 sec latency 17.476 ms oleg457-client.virtnet: 1 66222 1.53 MB/sec execute 191 sec latency 13.078 ms oleg457-client.virtnet: 1 66648 1.55 MB/sec execute 192 sec latency 16.137 ms oleg457-client.virtnet: 1 67363 1.57 MB/sec execute 193 sec latency 15.468 ms oleg457-client.virtnet: 1 67697 1.57 MB/sec execute 194 sec latency 15.916 ms oleg457-client.virtnet: 1 68150 1.57 MB/sec execute 195 sec latency 14.078 ms oleg457-client.virtnet: 1 68651 1.57 MB/sec execute 196 sec latency 14.053 ms oleg457-client.virtnet: 1 69406 1.58 MB/sec execute 197 sec latency 9.072 ms test_70b fail mds2 10 times Failing mds2 on oleg457-server Stopping /mnt/lustre-mds2 (opts:) on oleg457-server 04:26:03 (1713428763) shut down Failover mds2 to oleg457-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 04:26:18 (1713428778) targets are mounted 04:26:18 (1713428778) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1285344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2016 1285672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593108 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36608 3569876 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49552 7162984 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 04:26:31 (1713428791) shut down Failover mds1 to oleg457-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-client.virtnet: 1 69821 1.58 MB/sec execute 198 sec latency 446.218 ms oleg457-client.virtnet: 1 70757 1.61 MB/sec execute 199 sec latency 12.526 ms oleg457-client.virtnet: 1 71273 1.62 MB/sec execute 200 sec latency 12.792 ms oleg457-client.virtnet: 1 71349 1.61 MB/sec execute 201 sec latency 811.868 ms oleg457-client.virtnet: 1 71349 1.60 MB/sec execute 202 sec latency 1812.031 ms oleg457-client.virtnet: 1 71349 1.60 MB/sec execute 203 sec latency 2812.266 ms oleg457-client.virtnet: 1 71349 1.59 MB/sec execute 204 sec latency 3812.502 ms oleg457-client.virtnet: 1 71349 1.58 MB/sec execute 205 sec latency 4812.700 ms oleg457-client.virtnet: 1 71349 1.57 MB/sec execute 206 sec latency 5812.918 ms oleg457-client.virtnet: 1 71349 1.57 MB/sec execute 207 sec latency 6813.153 ms oleg457-client.virtnet: 1 71349 1.56 MB/sec execute 208 sec latency 7813.404 ms oleg457-client.virtnet: 1 71349 1.55 MB/sec execute 209 sec latency 8813.716 ms oleg457-client.virtnet: 1 71349 1.54 MB/sec execute 210 sec latency 9813.937 ms oleg457-client.virtnet: 1 71349 1.54 MB/sec execute 211 sec latency 10814.070 ms oleg457-client.virtnet: 1 71349 1.53 MB/sec execute 212 sec latency 11814.212 ms oleg457-client.virtnet: 1 71349 1.52 MB/sec execute 213 sec latency 12814.402 ms oleg457-client.virtnet: 1 71349 1.51 MB/sec execute 214 sec latency 13814.596 ms oleg457-client.virtnet: 1 71349 1.51 MB/sec execute 215 sec latency 14814.771 ms oleg457-client.virtnet: 1 71349 1.50 MB/sec execute 216 sec latency 15815.006 ms oleg457-client.virtnet: 1 71349 1.49 MB/sec execute 217 sec latency 16815.182 ms oleg457-client.virtnet: 1 71349 1.49 MB/sec execute 218 sec latency 17815.457 ms oleg457-client.virtnet: 1 71349 1.48 MB/sec execute 219 sec latency 18815.699 ms oleg457-client.virtnet: 1 71464 1.47 MB/sec execute 220 sec latency 19298.377 ms oleg457-client.virtnet: 1 71760 1.47 MB/sec execute 221 sec latency 15.909 ms oleg457-client.virtnet: 1 72056 1.47 MB/sec execute 222 sec latency 22.066 ms oleg457-client.virtnet: 1 72444 1.48 MB/sec execute 223 sec latency 15.457 ms oleg457-client.virtnet: 1 72815 1.47 MB/sec execute 224 sec latency 17.694 ms oleg457-client.virtnet: 1 73432 1.47 MB/sec execute 225 sec latency 11.944 ms oleg457-client.virtnet: 1 74202 1.49 MB/sec execute 226 sec latency 14.924 ms oleg457-client.virtnet: 1 74739 1.51 MB/sec execute 227 sec latency 12.132 ms oleg457-client.virtnet: 1 74977 1.51 MB/sec execute 228 sec latency 15.837 ms oleg457-client.virtnet: 1 75322 1.50 MB/sec execute 229 sec latency 23.281 ms oleg457-client.virtnet: 1 75687 1.50 MB/sec execute 230 sec latency 16.497 ms oleg457-client.virtnet: 1 76398 1.51 MB/sec execute 231 sec latency 16.715 ms oleg457-client.virtnet: 1 76940 1.51 MB/sec execute 232 sec latency 18.560 ms oleg457-client.virtnet: 1 77810 1.53 MB/sec execute 233 sec latency 12.309 ms oleg457-client.virtnet: 1 78536 1.55 MB/sec execute 234 sec latency 7.986 ms oleg457-client.virtnet: 1 79194 1.55 MB/sec execute 235 sec latency 11.399 ms oleg457-client.virtnet: 1 79916 1.56 MB/sec execute 236 sec latency 10.879 ms oleg457-client.virtnet: 1 80683 1.56 MB/sec execute 237 sec latency 9.337 ms oleg457-client.virtnet: 1 81779 1.60 MB/sec execute 238 sec latency 8.773 ms oleg457-client.virtnet: 1 82107 1.60 MB/sec execute 239 sec latency 15.617 ms oleg457-client.virtnet: 1 82635 1.59 MB/sec execute 240 sec latency 12.814 ms oleg457-client.virtnet: 1 83368 1.61 MB/sec execute 241 sec latency 9.327 ms oleg457-client.virtnet: 1 84104 1.6oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 04:26:45 (1713428805) targets are mounted 04:26:45 (1713428805) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 MB/sec execute 242 sec latency 7.937 ms oleg457-client.virtnet: 1 85286 1.64 MB/sec execute 243 sec latency 8.324 ms oleg457-client.virtnet: 1 85880 1.64 MB/sec execute 244 sec latency 10.202 ms oleg457-client.virtnet: 1 86608 1.66 MB/sec execute 245 sec latency 14.202 ms oleg457-client.virtnet: 1 87316 1.65 MB/sec execute 246 sec latency 8.339 ms oleg457-client.virtnet: 1 87785 1.65 MB/sec execute 247 sec latency 16.391 ms oleg457-client.virtnet: 1 88934 1.69 MB/sec execute 248 sec latency 7.993 ms oleg457-client.virtnet: 1 89318 1.69 MB/sec execute 249 sec latency 14.518 ms oleg457-client.virtnet: 1 89660 1.68 MB/sec execute 250 sec latency 19.100 ms oleg457-client.virtnet: 1 90118 1.69 MB/sec execute 251 sec latency 18.710 ms oleg457-client.virtnet: 1 90536 1.69 MB/sec execute 252 sec latency 14.709 ms oleg457-client.virtnet: 1 90979 1.68 MB/sec execute 253 sec latency 11.095 ms oleg457-client.virtnet: 1 91626 1.70 MB/sec execute 254 sec latency 16.564 ms oleg457-client.virtnet: 1 92259 1.71 MB/sec execute 255 sec latency 16.961 ms oleg457-client.virtnet: 1 92563 1.71 MB/sec execute 256 sec latency 16.683 ms oleg457-client.virtnet: 1 92956 1.71 MB/sec execute 257 sec latency 16.567 ms oleg457-client.virtnet: 1 93682 1.72 MB/sec execute 258 sec latency 8.588 ms oleg457-client.virtnet: 1 94236 1.72 MB/sec execute 259 sec latency 16.802 ms oleg457-client.virtnet: 1 94824 1.72 MB/sec execute 260 sec latency 11.263 ms oleg457-client.virtnet: 1 95802 1.75 MB/sec execute 261 sec latency 11.573 ms oleg457-client.virtnet: 1 96119 1.75 MB/sec execute 262 sec latency 15.730 ms oleg457-client.virtnet: 1 96568 1.74 MB/sec execute 263 sec latency 16.260 ms oleg457-client.virtnet: 1 97325 1.75 MB/sec execute 264 sec latency 8.165 ms oleg457-client.virtnet: 1 97876 1.75 MB/sec execute 265 sec latency 10.509 ms oleg457-client.virtnet: 1 98259 1.75 MB/sec execute 266 sec latency 15.540 ms oleg457-client.virtnet: 1 98827 1.76 MB/sec execute 267 sec latency 15.565 ms oleg457-client.virtnet: 1 99578 1.78 MB/sec execute 268 sec latency 14.556 ms oleg457-client.virtnet: 1 100065 1.77 MB/sec execute 269 sec latency 11.368 ms oleg457-client.virtnet: 1 100796 1.79 MB/sec execute 270 sec latency 8.832 ms oleg457-client.virtnet: 1 101548 1.78 MB/sec execute 271 sec latency 8.509 ms oleg457-client.virtnet: 1 102698 1.81 MB/sec execute 272 sec latency 7.624 ms oleg457-client.virtnet: 1 103347 1.82 MB/sec execute 273 sec latency 12.653 ms oleg457-client.virtnet: 1 103978 1.82 MB/sec execute 274 sec latency 11.089 ms oleg457-client.virtnet: 1 104677 1.82 MB/sec execute 275 sec latency 11.015 ms oleg457-client.virtnet: 1 105418 1.83 MB/sec execute 276 sec latency 8.348 ms oleg457-client.virtnet: 1 106585 1.86 MB/sec execute 277 sec latency 8.258 ms oleg457-client.virtnet: 1 107161 1.85 MB/sec execute 278 sec latency 11.422 ms oleg457-client.virtnet: 1 107858 1.87 MB/sec execute 279 sec latency 9.968 ms oleg457-client.virtnet: 1 108301 1.86 MB/sec execute 280 sec latency 13.428 ms oleg457-client.virtnet: 1 109036 1.86 MB/sec execute 281 sec latency 11.278 ms oleg457-client.virtnet: 1 110165 1.89 MB/sec execute 282 sec latency 12.862 ms oleg457-client.virtnet: 1 110663 1.89 MB/sec execute 283 sec latency 15.181 ms oleg457-client.virtnet: 1 111325 1.89 MB/sec execute 284 sec latency 11.052 ms oleg457-client.virtnet: 1 112083 1.90 MB/sec execute 285 sec latency 11.952 ms oleg457-client.virtnet: 1 113015 1.91 MB/sec execute 286 sec latency 8.926 ms oleg457-client.virtnet: 1 113937 1.93 MB/sec execute 287 sec latency 7.975 ms oleg457-client.virtnet: 1 114602 1.93 MB/sec execute 288 sec latency 11.246 ms oleg457-client.virtnet: 1 115371 1.93 MB/sec execute 289 sec latency 10.438 ms oleg457-client.virtnet: 1 116114 1.94 MB/sec execute 290 sec latency 9.886 ms oleg457-client.virtnet: 1 117209 1.96 MB/sec execute 291 sec latency 9.365 ms oleg457-client.virtnet: 1 117800 1.96 MB/sec execute 292 sec latency 9.319 ms oleg457-client.virtnet: 1 118535 1.97 MB/sec execute 293 sec latency 8.585 ms oleg457-client.virtnet: 1 119274 1.97 MB/sec execute 294 sec latency 8.670 ms oleg457-client.virtnet: 1 120404 1.99 MB/sec execute 295 sec latency 7.741 ms oleg457-client.virtnet: 1 121091 2.00 MB/sec execute 296 sec latency 8.267 ms oleg457-client.virtnet: 1 121777 2.00 MB/sec execute 297 sec latency 11.858 ms oleg457-client.virtnet: 1 122473 2.00 MB/sec execute 298 sec latency 9.077 ms oleg457-client.virtnet: 1 123350 2.02 MB/sec execute 299 sec latency 7.894 ms oleg457-client.virtnet: 1 cleanup 300 sec oleg457-client.virtnet: 0 cleanup 300 sec oleg457-client.virtnet: oleg457-client.virtnet: Operation Count AvgLat MaxLat oleg457-client.virtnet: ---------------------------------------- oleg457-client.virtnet: NTCreateX 19450 9.238 19837.130 oleg457-client.virtnet: Close 14321 2.746 19946.936 oleg457-client.virtnet: Rename 828 9.021 23.260 oleg457-client.virtnet: Unlink 3901 2.375 8.977 oleg457-client.virtnet: Qpathinfo 17676 1.697 18.294 oleg457-client.virtnet: Qfileinfo 3094 0.171 3.104 oleg457-client.virtnet: Qfsinfo 3239 0.285 11.661 oleg457-client.virtnet: Sfileinfo 1582 4.746 15.742 oleg457-client.virtnet: Find 6835 0.487 17.691 oleg457-client.virtnet: WriteX 9646 1.089 13.812 oleg457-client.virtnet: ReadX 30719 0.021 2.221 oleg457-client.virtnet: LockX 64 1.427 2.290 oleg457-client.virtnet: UnlockX 64 1.450 2.123 oleg457-client.virtnet: Flush 1352 7.247 584.647 oleg457-client.virtnet: oleg457-client.virtnet: Throughput 2.01605 MB/sec 1 clients 1 procs max_latency=19946.969 ms oleg457-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg457-client.virtnet at Thu Apr 18 04:27:43 EDT 2024 with return code 0 oleg457-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg457-client.virtnet oleg457-client.virtnet: /mnt/lustre/d70b.replay-single/oleg457-client.virtnet /mnt/lustre/d70b.replay-single/oleg457-client.virtnet oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg457-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg457-client.virtnet: removed directory: 'clients/client0' oleg457-client.virtnet: removed directory: 'clients' oleg457-client.virtnet: removed 'client.txt' oleg457-client.virtnet: /mnt/lustre/d70b.replay-single/oleg457-client.virtnet oleg457-client.virtnet: dbench successfully finished