== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 11:48:18 (1713282498) Starting client oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre Started clients oleg127-client.virtnet: 192.168.201.127@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg127-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=21704 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2160 1285528 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3580864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1640 3594096 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27796 7174960 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 11:48:27 (1713282507) shut down Failover mds1 to oleg127-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 11:48:42 (1713282522) targets are mounted 11:48:42 (1713282522) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2128 1285560 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26412 3568868 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1792 3594192 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28204 7163060 1% /mnt/lustre oleg127-client.virtnet: looking for dbench program oleg127-client.virtnet: /usr/bin/dbench oleg127-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg127-client.virtnet oleg127-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg127-client.virtnet' oleg127-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg127-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg127-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg127-client.virtnet at Tue Apr 16 11:48:19 EDT 2024 oleg127-client.virtnet: waiting for dbench pid 21728 oleg127-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg127-client.virtnet: oleg127-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg127-client.virtnet: failed to create barrier semaphore oleg127-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg127-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg127-client.virtnet: releasing clients oleg127-client.virtnet: 1 313 11.48 MB/sec warmup 1 sec latency 21.381 ms oleg127-client.virtnet: 1 673 10.59 MB/sec warmup 2 sec latency 18.102 ms oleg127-client.virtnet: 1 977 7.30 MB/sec warmup 3 sec latency 16.990 ms oleg127-client.virtnet: 1 1448 6.50 MB/sec warmup 4 sec latency 15.101 ms oleg127-client.virtnet: 1 1808 5.27 MB/sec warmup 5 sec latency 17.848 ms oleg127-client.virtnet: 1 2184 4.46 MB/sec warmup 6 sec latency 10.815 ms oleg127-client.virtnet: 1 2596 4.14 MB/sec warmup 7 sec latency 15.604 ms oleg127-client.virtnet: 1 3352 4.42 MB/sec warmup 8 sec latency 26.091 ms oleg127-client.virtnet: 1 3996 4.45 MB/sec warmup 9 sec latency 12.234 ms oleg127-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 836.363 ms oleg127-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 1836.624 ms oleg127-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 2836.885 ms oleg127-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 3837.136 ms oleg127-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 4837.409 ms oleg127-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 5837.654 ms oleg127-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 6837.899 ms oleg127-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 7838.136 ms oleg127-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 8838.365 ms oleg127-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 9838.594 ms oleg127-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 10838.802 ms oleg127-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 11839.089 ms oleg127-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 12839.266 ms oleg127-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 13839.488 ms oleg127-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 14839.727 ms oleg127-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 15839.968 ms oleg127-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 16840.200 ms oleg127-client.virtnet: 1 4268 1.49 MB/sec warmup 27 sec latency 17500.659 ms oleg127-client.virtnet: 1 4606 1.50 MB/sec warmup 28 sec latency 16.276 ms oleg127-client.virtnet: 1 5021 1.56 MB/sec warmup 29 sec latency 15.571 ms oleg127-client.virtnet: 1 5334 1.51 MB/sec warmup 30 sec latency 17.735 ms oleg127-client.virtnet: 1 5726 1.48 MB/sec warmup 31 sec latency 12.126 ms oleg127-client.virtnet: 1 6135 1.50 MB/sec warmup 32 sec latency 19.143 ms oleg127-client.virtnet: 1 6436 1.55 MB/sec warmup 33 sec latency 650.103test_70b fail mds2 2 times Failing mds2 on oleg127-server Stopping /mnt/lustre-mds2 (opts:) on oleg127-server 11:48:55 (1713282535) shut down Failover mds2 to oleg127-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0001 11:49:11 (1713282551) targets are mounted 11:49:11 (1713282551) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569476 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12688 3594148 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50112 7163624 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 11:49:24 (1713282564) shut down Failover mds1 to oleg127-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 ms oleg127-client.virtnet: 1 7092 1.69 MB/sec warmup 34 sec latency 15.999 ms oleg127-client.virtnet: 1 7404 1.68 MB/sec warmup 35 sec latency 22.782 ms oleg127-client.virtnet: 1 7542 1.64 MB/sec warmup 36 sec latency 454.362 ms oleg127-client.virtnet: 1 7542 1.60 MB/sec warmup 37 sec latency 1454.607 ms oleg127-client.virtnet: 1 7542 1.56 MB/sec warmup 38 sec latency 2454.809 ms oleg127-client.virtnet: 1 7542 1.52 MB/sec warmup 39 sec latency 3455.123 ms oleg127-client.virtnet: 1 7542 1.48 MB/sec warmup 40 sec latency 4455.342 ms oleg127-client.virtnet: 1 7542 1.44 MB/sec warmup 41 sec latency 5455.549 ms oleg127-client.virtnet: 1 7542 1.41 MB/sec warmup 42 sec latency 6455.792 ms oleg127-client.virtnet: 1 7542 1.38 MB/sec warmup 43 sec latency 7456.024 ms oleg127-client.virtnet: 1 7542 1.34 MB/sec warmup 44 sec latency 8456.258 ms oleg127-client.virtnet: 1 7542 1.31 MB/sec warmup 45 sec latency 9456.518 ms oleg127-client.virtnet: 1 7542 1.29 MB/sec warmup 46 sec latency 10456.767 ms oleg127-client.virtnet: 1 7542 1.26 MB/sec warmup 47 sec latency 11456.996 ms oleg127-client.virtnet: 1 7542 1.23 MB/sec warmup 48 sec latency 12457.186 ms oleg127-client.virtnet: 1 7542 1.21 MB/sec warmup 49 sec latency 13457.375 ms oleg127-client.virtnet: 1 7542 1.18 MB/sec warmup 50 sec latency 14457.534 ms oleg127-client.virtnet: 1 7542 1.16 MB/sec warmup 51 sec latency 15457.688 ms oleg127-client.virtnet: 1 7542 1.14 MB/sec warmup 52 sec latency 16457.889 ms oleg127-client.virtnet: 1 7542 1.12 MB/sec warmup 53 sec latency 17458.059 ms oleg127-client.virtnet: 1 7542 1.10 MB/sec warmup 54 sec latency 18458.232 ms oleg127-client.virtnet: 1 7542 1.08 MB/sec warmup 55 sec latency 19458.552 ms oleg127-client.virtnet: 1 7701 1.06 MB/sec warmup 56 sec latency 19880.063 ms oleg127-client.virtnet: 1 8030 1.05 MB/sec warmup 57 sec latency 25.474 ms oleg127-client.virtnet: 1 8371 1.06 MB/sec warmup 58 sec latency 16.907 ms oleg127-client.virtnet: 1 8761 1.09 MB/sec warmup 59 sec latency 15.225 ms oleg127-client.virtnet: 1 9532 1.58 MB/sec execute 1 sec latency 16.516 ms oleg127-client.virtnet: 1 10100 3.50 MB/sec execute 2 sec latency 15.748 ms oleg127-client.virtnet: 1 10732 4.03 MB/sec execute 3 sec latency 14.767 ms oleg127-client.virtnet: 1 11021 3.38 MB/sec execute 4 sec latency 15.288 ms oleg127-client.virtnet: 1 11359 2.76 MB/sec execute 5 sec latency 21.539 ms oleg127-client.virtnet: 1 11653 2.42 MB/sec execute 6 sec latency 15.499 ms oleg127-client.virtnet: 1 12121 2.66 MB/sec execute 7 sec latency 15.568 ms oleg127-client.virtnet: 1 12444 2.36 MB/sec execute 8 sec latency 16.361 ms oleg127-client.virtnet: 1 12836 2.15 MB/sec execute 9 sec latency 10.921 ms oleg127-client.virtnet: 1 13255 2.16 MB/sec execute 10 sec latency 15.063 ms oleg127-client.virtnet: 1 14004 2.54 MB/sec execute 11 sec latency 14.957 ms oleg127-client.virtnet: 1 14436 2.70 MB/sec execute 12 sec latency 14.612 ms oleg127-client.virtnet: 1 14673 2.51 MB/sec execute 13 sec latency 15.222 ms oleg127-client.virtnet: 1 15051 2.38 MB/sec execute 14 sec latency 17.582 ms oleg127-client.virtnet: 1 15404 2.32 MB/sec execute 15 sec latency 15.187 ms oleg127-client.virtnet: 1 16033 2.38 MB/sec execute 16 sec latency 15.009 ms oleg127-client.virtnet: 1 16634 2.34 MB/sec execute 17 sec latency 14.826 ms oleg127-client.virtnet: 1 17205 2.51 MB/sec execute 18 sec latency 14.910 ms oleg127-client.virtnet: 1 17876 2.66 MB/sec executoleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 11:49:39 (1713282579) targets are mounted 11:49:39 (1713282579) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37548 3568884 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12064 3591904 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7160788 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg127-server Stopping /mnt/lustre-mds2 (opts:) on oleg127-server 11:49:52 (1713282592) shut down Failover mds2 to oleg127-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0001 11:50:07 (1713282607) targets are mounted 11:50:07 (1713282607) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37512 3568732 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12192 3591824 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49704 7160556 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 11:50:21 (1713282621) shut down e 19 sec latency 11.905 ms oleg127-client.virtnet: 1 18162 2.59 MB/sec execute 20 sec latency 15.832 ms oleg127-client.virtnet: 1 18486 2.48 MB/sec execute 21 sec latency 22.168 ms oleg127-client.virtnet: 1 18816 2.44 MB/sec execute 22 sec latency 16.445 ms oleg127-client.virtnet: 1 19233 2.47 MB/sec execute 23 sec latency 15.143 ms oleg127-client.virtnet: 1 19568 2.38 MB/sec execute 24 sec latency 16.698 ms oleg127-client.virtnet: 1 19974 2.31 MB/sec execute 25 sec latency 10.116 ms oleg127-client.virtnet: 1 20377 2.30 MB/sec execute 26 sec latency 15.245 ms oleg127-client.virtnet: 1 21132 2.45 MB/sec execute 27 sec latency 15.249 ms oleg127-client.virtnet: 1 21545 2.52 MB/sec execute 28 sec latency 16.175 ms oleg127-client.virtnet: 1 21775 2.45 MB/sec execute 29 sec latency 16.671 ms oleg127-client.virtnet: 1 22114 2.38 MB/sec execute 30 sec latency 17.153 ms oleg127-client.virtnet: 1 22432 2.35 MB/sec execute 31 sec latency 16.990 ms oleg127-client.virtnet: 1 22827 2.38 MB/sec execute 32 sec latency 16.672 ms oleg127-client.virtnet: 1 22854 2.31 MB/sec execute 33 sec latency 923.058 ms oleg127-client.virtnet: 1 22854 2.24 MB/sec execute 34 sec latency 1923.339 ms oleg127-client.virtnet: 1 22854 2.17 MB/sec execute 35 sec latency 2923.598 ms oleg127-client.virtnet: 1 22854 2.11 MB/sec execute 36 sec latency 3923.989 ms oleg127-client.virtnet: 1 22854 2.06 MB/sec execute 37 sec latency 4924.400 ms oleg127-client.virtnet: 1 22854 2.00 MB/sec execute 38 sec latency 5924.697 ms oleg127-client.virtnet: 1 22854 1.95 MB/sec execute 39 sec latency 6924.995 ms oleg127-client.virtnet: 1 22854 1.90 MB/sec execute 40 sec latency 7925.383 ms oleg127-client.virtnet: 1 22854 1.86 MB/sec execute 41 sec latency 8925.662 ms oleg127-client.virtnet: 1 22854 1.81 MB/sec execute 42 sec latency 9925.961 ms oleg127-client.virtnet: 1 22854 1.77 MB/sec execute 43 sec latency 10926.164 ms oleg127-client.virtnet: 1 22854 1.73 MB/sec execute 44 sec latency 11926.325 ms oleg127-client.virtnet: 1 22854 1.69 MB/sec execute 45 sec latency 12926.498 ms oleg127-client.virtnet: 1 22854 1.65 MB/sec execute 46 sec latency 13926.641 ms oleg127-client.virtnet: 1 22854 1.62 MB/sec execute 47 sec latency 14926.838 ms oleg127-client.virtnet: 1 22854 1.59 MB/sec execute 48 sec latency 15927.164 ms oleg127-client.virtnet: 1 22854 1.55 MB/sec execute 49 sec latency 16927.444 ms oleg127-client.virtnet: 1 22854 1.52 MB/sec execute 50 sec latency 17927.623 ms oleg127-client.virtnet: 1 22854 1.49 MB/sec execute 51 sec latency 18927.868 ms oleg127-client.virtnet: 1 22854 1.46 MB/sec execute 52 sec latency 19928.104 ms oleg127-client.virtnet: 1 23191 1.44 MB/sec execute 53 sec latency 19930.486 ms oleg127-client.virtnet: 1 23615 1.44 MB/sec execute 54 sec latency 14.234 ms oleg127-client.virtnet: 1 23958 1.44 MB/sec execute 55 sec latency 16.105 ms oleg127-client.virtnet: 1 24675 1.52 MB/sec execute 56 sec latency 15.110 ms oleg127-client.virtnet: 1 25099 1.57 MB/sec execute 57 sec latency 15.947 ms oleg127-client.virtnet: 1 25333 1.55 MB/sec execute 58 sec latency 15.505 ms oleg127-client.virtnet: 1 25665 1.53 MB/sec execute 59 sec latency 20.964 ms oleg127-client.virtnet: 1 26014 1.54 MB/sec execute 60 sec latency 15.374 ms oleg127-client.virtnet: 1 26441 1.56 MB/sec execute 61 sec latency 16.377 ms oleg127-client.virtnet: 1 26794 1.54 MB/sec execute 62 sec latency 15.978 ms oleg127-client.virtnet: 1 27224 1.54 MB/sec execute 63 sec latency 26.212 ms oleg127-cliFailover mds1 to oleg127-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 11:50:36 (1713282636) targets are mounted 11:50:36 (1713282636) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37560 3568872 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12064 3591904 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160776 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg127-server Stopping /mnt/lustre-mds2 (opts:) on oleg127-server 11:50:49 (1713282649) shut down Failover mds2 to oleg127-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0001 11:51:05 (1713282665) targets are mounted 11:51:05 (1713282665) facet_failover done ent.virtnet: 1 27651 1.58 MB/sec execute 64 sec latency 15.625 ms oleg127-client.virtnet: 1 28363 1.65 MB/sec execute 65 sec latency 15.093 ms oleg127-client.virtnet: 1 28712 1.65 MB/sec execute 66 sec latency 15.544 ms oleg127-client.virtnet: 1 28974 1.63 MB/sec execute 67 sec latency 15.551 ms oleg127-client.virtnet: 1 29324 1.62 MB/sec execute 68 sec latency 15.859 ms oleg127-client.virtnet: 1 29667 1.61 MB/sec execute 69 sec latency 15.133 ms oleg127-client.virtnet: 1 30069 1.63 MB/sec execute 70 sec latency 15.545 ms oleg127-client.virtnet: 1 30419 1.62 MB/sec execute 71 sec latency 16.846 ms oleg127-client.virtnet: 1 30820 1.61 MB/sec execute 72 sec latency 15.129 ms oleg127-client.virtnet: 1 31346 1.67 MB/sec execute 73 sec latency 15.784 ms oleg127-client.virtnet: 1 31987 1.71 MB/sec execute 74 sec latency 14.760 ms oleg127-client.virtnet: 1 32284 1.71 MB/sec execute 75 sec latency 16.185 ms oleg127-client.virtnet: 1 32585 1.69 MB/sec execute 76 sec latency 18.133 ms oleg127-client.virtnet: 1 32896 1.68 MB/sec execute 77 sec latency 17.706 ms oleg127-client.virtnet: 1 33239 1.67 MB/sec execute 78 sec latency 16.163 ms oleg127-client.virtnet: 1 33637 1.69 MB/sec execute 79 sec latency 16.107 ms oleg127-client.virtnet: 1 33984 1.67 MB/sec execute 80 sec latency 18.419 ms oleg127-client.virtnet: 1 34383 1.67 MB/sec execute 81 sec latency 15.768 ms oleg127-client.virtnet: 1 34896 1.72 MB/sec execute 82 sec latency 16.011 ms oleg127-client.virtnet: 1 35527 1.76 MB/sec execute 83 sec latency 15.460 ms oleg127-client.virtnet: 1 35818 1.75 MB/sec execute 84 sec latency 15.919 ms oleg127-client.virtnet: 1 36099 1.73 MB/sec execute 85 sec latency 16.870 ms oleg127-client.virtnet: 1 36435 1.72 MB/sec execute 86 sec latency 15.535 ms oleg127-client.virtnet: 1 36584 1.72 MB/sec execute 87 sec latency 587.130 ms oleg127-client.virtnet: 1 36995 1.73 MB/sec execute 88 sec latency 16.125 ms oleg127-client.virtnet: 1 37321 1.72 MB/sec execute 89 sec latency 16.007 ms oleg127-client.virtnet: 1 37520 1.70 MB/sec execute 90 sec latency 511.023 ms oleg127-client.virtnet: 1 37520 1.68 MB/sec execute 91 sec latency 1511.271 ms oleg127-client.virtnet: 1 37520 1.66 MB/sec execute 92 sec latency 2511.656 ms oleg127-client.virtnet: 1 37520 1.64 MB/sec execute 93 sec latency 3512.034 ms oleg127-client.virtnet: 1 37520 1.63 MB/sec execute 94 sec latency 4512.304 ms oleg127-client.virtnet: 1 37520 1.61 MB/sec execute 95 sec latency 5512.619 ms oleg127-client.virtnet: 1 37520 1.59 MB/sec execute 96 sec latency 6512.929 ms oleg127-client.virtnet: 1 37520 1.58 MB/sec execute 97 sec latency 7513.162 ms oleg127-client.virtnet: 1 37520 1.56 MB/sec execute 98 sec latency 8513.404 ms oleg127-client.virtnet: 1 37520 1.55 MB/sec execute 99 sec latency 9513.755 ms oleg127-client.virtnet: 1 37520 1.53 MB/sec execute 100 sec latency 10514.040 ms oleg127-client.virtnet: 1 37520 1.51 MB/sec execute 101 sec latency 11514.277 ms oleg127-client.virtnet: 1 37520 1.50 MB/sec execute 102 sec latency 12514.579 ms oleg127-client.virtnet: 1 37520 1.49 MB/sec execute 103 sec latency 13514.810 ms oleg127-client.virtnet: 1 37520 1.47 MB/sec execute 104 sec latency 14515.024 ms oleg127-client.virtnet: 1 37520 1.46 MB/sec execute 105 sec latency 15515.304 ms oleg127-client.virtnet: 1 37520 1.44 MB/sec execute 106 sec latency 16515.618 ms oleg127-client.virtnet: 1 37520 1.43 MB/sec execute 107 sec latency 17515.863 ms oleg127-client.virtnet: 1 37520 1.42 MB/sec exeoleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37540 3568776 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12084 3592008 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160784 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 11:51:18 (1713282678) shut down Failover mds1 to oleg127-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 11:51:33 (1713282693) targets are mounted 11:51:33 (1713282693) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37560 3568696 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12064 3593324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7162020 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg127-server Stopping /mnt/lustre-mds2 (opts:) on oleg127-server 11:51:46 (1713282706) shut down cute 108 sec latency 18516.149 ms oleg127-client.virtnet: 1 37520 1.40 MB/sec execute 109 sec latency 19516.512 ms oleg127-client.virtnet: 1 37851 1.40 MB/sec execute 110 sec latency 19673.049 ms oleg127-client.virtnet: 1 38273 1.43 MB/sec execute 111 sec latency 16.021 ms oleg127-client.virtnet: 1 38878 1.45 MB/sec execute 112 sec latency 15.903 ms oleg127-client.virtnet: 1 39334 1.47 MB/sec execute 113 sec latency 15.153 ms oleg127-client.virtnet: 1 39571 1.46 MB/sec execute 114 sec latency 17.196 ms oleg127-client.virtnet: 1 39914 1.45 MB/sec execute 115 sec latency 19.365 ms oleg127-client.virtnet: 1 40235 1.45 MB/sec execute 116 sec latency 15.580 ms oleg127-client.virtnet: 1 40624 1.47 MB/sec execute 117 sec latency 15.597 ms oleg127-client.virtnet: 1 40987 1.46 MB/sec execute 118 sec latency 18.718 ms oleg127-client.virtnet: 1 41639 1.47 MB/sec execute 119 sec latency 14.411 ms oleg127-client.virtnet: 1 42358 1.51 MB/sec execute 120 sec latency 14.841 ms oleg127-client.virtnet: 1 42821 1.53 MB/sec execute 121 sec latency 15.033 ms oleg127-client.virtnet: 1 43060 1.52 MB/sec execute 122 sec latency 15.370 ms oleg127-client.virtnet: 1 43500 1.52 MB/sec execute 123 sec latency 16.031 ms oleg127-client.virtnet: 1 43835 1.51 MB/sec execute 124 sec latency 15.263 ms oleg127-client.virtnet: 1 44246 1.53 MB/sec execute 125 sec latency 16.058 ms oleg127-client.virtnet: 1 44605 1.52 MB/sec execute 126 sec latency 17.050 ms oleg127-client.virtnet: 1 45011 1.52 MB/sec execute 127 sec latency 15.000 ms oleg127-client.virtnet: 1 45570 1.55 MB/sec execute 128 sec latency 15.492 ms oleg127-client.virtnet: 1 46207 1.58 MB/sec execute 129 sec latency 14.169 ms oleg127-client.virtnet: 1 46495 1.57 MB/sec execute 130 sec latency 15.567 ms oleg127-client.virtnet: 1 46816 1.56 MB/sec execute 131 sec latency 15.034 ms oleg127-client.virtnet: 1 47126 1.56 MB/sec execute 132 sec latency 14.963 ms oleg127-client.virtnet: 1 47584 1.58 MB/sec execute 133 sec latency 14.693 ms oleg127-client.virtnet: 1 47932 1.57 MB/sec execute 134 sec latency 16.063 ms oleg127-client.virtnet: 1 48312 1.56 MB/sec execute 135 sec latency 10.972 ms oleg127-client.virtnet: 1 48707 1.56 MB/sec execute 136 sec latency 16.019 ms oleg127-client.virtnet: 1 49352 1.59 MB/sec execute 137 sec latency 16.422 ms oleg127-client.virtnet: 1 49838 1.61 MB/sec execute 138 sec latency 12.986 ms oleg127-client.virtnet: 1 50103 1.61 MB/sec execute 139 sec latency 19.822 ms oleg127-client.virtnet: 1 50433 1.60 MB/sec execute 140 sec latency 21.799 ms oleg127-client.virtnet: 1 50747 1.60 MB/sec execute 141 sec latency 16.060 ms oleg127-client.virtnet: 1 51152 1.61 MB/sec execute 142 sec latency 15.890 ms oleg127-client.virtnet: 1 51479 1.60 MB/sec execute 143 sec latency 17.104 ms oleg127-client.virtnet: 1 51662 1.59 MB/sec execute 144 sec latency 571.063 ms oleg127-client.virtnet: 1 52073 1.59 MB/sec execute 145 sec latency 11.191 ms oleg127-client.virtnet: 1 52596 1.62 MB/sec execute 146 sec latency 16.069 ms oleg127-client.virtnet: 1 52933 1.62 MB/sec execute 147 sec latency 484.711 ms oleg127-client.virtnet: 1 52933 1.61 MB/sec execute 148 sec latency 1484.967 ms oleg127-client.virtnet: 1 52933 1.60 MB/sec execute 149 sec latency 2485.357 ms oleg127-client.virtnet: 1 52933 1.58 MB/sec execute 150 sec latency 3485.644 ms oleg127-client.virtnet: 1 52933 1.57 MB/sec execute 151 sec latency 4485.935 ms oleg127-client.virtnet: 1 52933 1.56 MB/sec execute 152 sec latency 5486.188 ms oleg127-client.virtnet: 1 52933 Failover mds2 to oleg127-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0001 11:52:02 (1713282722) targets are mounted 11:52:02 (1713282722) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37548 3568796 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12064 3591872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7160668 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 11:52:15 (1713282735) shut down Failover mds1 to oleg127-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 11:52:31 (1713282751) targets are mounted 11:52:31 (1713282751) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1.55 MB/sec execute 153 sec latency 6486.432 ms oleg127-client.virtnet: 1 52933 1.54 MB/sec execute 154 sec latency 7486.648 ms oleg127-client.virtnet: 1 52933 1.53 MB/sec execute 155 sec latency 8486.910 ms oleg127-client.virtnet: 1 52933 1.52 MB/sec execute 156 sec latency 9487.246 ms oleg127-client.virtnet: 1 52933 1.51 MB/sec execute 157 sec latency 10487.516 ms oleg127-client.virtnet: 1 52933 1.50 MB/sec execute 158 sec latency 11487.791 ms oleg127-client.virtnet: 1 52933 1.49 MB/sec execute 159 sec latency 12488.067 ms oleg127-client.virtnet: 1 52933 1.49 MB/sec execute 160 sec latency 13488.289 ms oleg127-client.virtnet: 1 52933 1.48 MB/sec execute 161 sec latency 14488.512 ms oleg127-client.virtnet: 1 52933 1.47 MB/sec execute 162 sec latency 15488.749 ms oleg127-client.virtnet: 1 52933 1.46 MB/sec execute 163 sec latency 16489.067 ms oleg127-client.virtnet: 1 52933 1.45 MB/sec execute 164 sec latency 17489.220 ms oleg127-client.virtnet: 1 52933 1.44 MB/sec execute 165 sec latency 18489.426 ms oleg127-client.virtnet: 1 52933 1.43 MB/sec execute 166 sec latency 19489.639 ms oleg127-client.virtnet: 1 53342 1.45 MB/sec execute 167 sec latency 19701.053 ms oleg127-client.virtnet: 1 53622 1.45 MB/sec execute 168 sec latency 15.648 ms oleg127-client.virtnet: 1 53889 1.44 MB/sec execute 169 sec latency 14.966 ms oleg127-client.virtnet: 1 54168 1.43 MB/sec execute 170 sec latency 20.259 ms oleg127-client.virtnet: 1 54511 1.43 MB/sec execute 171 sec latency 15.648 ms oleg127-client.virtnet: 1 54902 1.44 MB/sec execute 172 sec latency 15.009 ms oleg127-client.virtnet: 1 55302 1.44 MB/sec execute 173 sec latency 16.189 ms oleg127-client.virtnet: 1 55692 1.44 MB/sec execute 174 sec latency 14.948 ms oleg127-client.virtnet: 1 56254 1.46 MB/sec execute 175 sec latency 17.661 ms oleg127-client.virtnet: 1 57056 1.49 MB/sec execute 176 sec latency 23.550 ms oleg127-client.virtnet: 1 57673 1.48 MB/sec execute 177 sec latency 11.272 ms oleg127-client.virtnet: 1 58441 1.50 MB/sec execute 178 sec latency 8.588 ms oleg127-client.virtnet: 1 59209 1.50 MB/sec execute 179 sec latency 8.693 ms oleg127-client.virtnet: 1 60370 1.55 MB/sec execute 180 sec latency 8.043 ms oleg127-client.virtnet: 1 60796 1.55 MB/sec execute 181 sec latency 11.077 ms oleg127-client.virtnet: 1 61242 1.55 MB/sec execute 182 sec latency 15.650 ms oleg127-client.virtnet: 1 61579 1.55 MB/sec execute 183 sec latency 15.209 ms oleg127-client.virtnet: 1 61978 1.56 MB/sec execute 184 sec latency 15.254 ms oleg127-client.virtnet: 1 62340 1.55 MB/sec execute 185 sec latency 16.962 ms oleg127-client.virtnet: 1 62748 1.55 MB/sec execute 186 sec latency 16.835 ms oleg127-client.virtnet: 1 63310 1.57 MB/sec execute 187 sec latency 15.629 ms oleg127-client.virtnet: 1 63944 1.59 MB/sec execute 188 sec latency 15.330 ms oleg127-client.virtnet: 1 64234 1.59 MB/sec execute 189 sec latency 15.083 ms oleg127-client.virtnet: 1 64546 1.58 MB/sec execute 190 sec latency 15.531 ms oleg127-client.virtnet: 1 64842 1.58 MB/sec execute 191 sec latency 15.488 ms oleg127-client.virtnet: 1 65272 1.59 MB/sec execute 192 sec latency 16.170 ms oleg127-client.virtnet: 1 65584 1.58 MB/sec execute 193 sec latency 17.133 ms oleg127-client.virtnet: 1 65942 1.58 MB/sec execute 194 sec latency 12.480 ms oleg127-client.virtnet: 1 66325 1.58 MB/sec execute 195 sec latency 15.738 ms oleg127-client.virtnet: 1 66885 1.60 MB/sec execute 196 sec latency 15.549 ms oleg127-client.virtnet: 1 67528 1.61 MB/sec execute 197 sec latency 15.407 ms olUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37608 3566480 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12028 3594000 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160480 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg127-server Stopping /mnt/lustre-mds2 (opts:) on oleg127-server 11:52:50 (1713282770) shut down Failover mds2 to oleg127-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0001 11:53:05 (1713282785) targets are mounted 11:53:05 (1713282785) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37596 3566668 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12028 3594828 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7161496 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 11:53:18 (1713282798) shut down eg127-client.virtnet: 1 67805 1.61 MB/sec execute 198 sec latency 18.060 ms oleg127-client.virtnet: 1 68121 1.61 MB/sec execute 199 sec latency 15.536 ms oleg127-client.virtnet: 1 68418 1.60 MB/sec execute 200 sec latency 16.236 ms oleg127-client.virtnet: 1 68867 1.61 MB/sec execute 201 sec latency 14.888 ms oleg127-client.virtnet: 1 69196 1.61 MB/sec execute 202 sec latency 17.121 ms oleg127-client.virtnet: 1 69585 1.60 MB/sec execute 203 sec latency 11.745 ms oleg127-client.virtnet: 1 69850 1.60 MB/sec execute 204 sec latency 337.134 ms oleg127-client.virtnet: 1 69850 1.59 MB/sec execute 205 sec latency 1337.320 ms oleg127-client.virtnet: 1 69850 1.59 MB/sec execute 206 sec latency 2337.509 ms oleg127-client.virtnet: 1 69850 1.58 MB/sec execute 207 sec latency 3337.750 ms oleg127-client.virtnet: 1 69850 1.57 MB/sec execute 208 sec latency 4337.956 ms oleg127-client.virtnet: 1 69850 1.56 MB/sec execute 209 sec latency 5338.177 ms oleg127-client.virtnet: 1 69850 1.56 MB/sec execute 210 sec latency 6338.404 ms oleg127-client.virtnet: 1 69850 1.55 MB/sec execute 211 sec latency 7338.632 ms oleg127-client.virtnet: 1 69850 1.54 MB/sec execute 212 sec latency 8338.821 ms oleg127-client.virtnet: 1 69850 1.53 MB/sec execute 213 sec latency 9339.041 ms oleg127-client.virtnet: 1 69850 1.53 MB/sec execute 214 sec latency 10339.322 ms oleg127-client.virtnet: 1 69850 1.52 MB/sec execute 215 sec latency 11339.609 ms oleg127-client.virtnet: 1 69850 1.51 MB/sec execute 216 sec latency 12339.955 ms oleg127-client.virtnet: 1 69850 1.51 MB/sec execute 217 sec latency 13340.198 ms oleg127-client.virtnet: 1 69850 1.50 MB/sec execute 218 sec latency 14340.481 ms oleg127-client.virtnet: 1 69850 1.49 MB/sec execute 219 sec latency 15340.631 ms oleg127-client.virtnet: 1 69850 1.48 MB/sec execute 220 sec latency 16340.830 ms oleg127-client.virtnet: 1 69850 1.48 MB/sec execute 221 sec latency 17341.018 ms oleg127-client.virtnet: 1 69850 1.47 MB/sec execute 222 sec latency 18341.237 ms oleg127-client.virtnet: 1 69850 1.46 MB/sec execute 223 sec latency 19341.465 ms oleg127-client.virtnet: 1 69850 1.46 MB/sec execute 224 sec latency 20341.598 ms oleg127-client.virtnet: 1 69850 1.45 MB/sec execute 225 sec latency 21341.774 ms oleg127-client.virtnet: 1 69850 1.45 MB/sec execute 226 sec latency 22341.977 ms oleg127-client.virtnet: 1 69850 1.44 MB/sec execute 227 sec latency 23342.174 ms oleg127-client.virtnet: 1 69850 1.43 MB/sec execute 228 sec latency 24342.361 ms oleg127-client.virtnet: 1 69850 1.43 MB/sec execute 229 sec latency 25342.553 ms oleg127-client.virtnet: 1 70213 1.44 MB/sec execute 230 sec latency 25699.135 ms oleg127-client.virtnet: 1 70923 1.46 MB/sec execute 231 sec latency 20.422 ms oleg127-client.virtnet: 1 71234 1.46 MB/sec execute 232 sec latency 15.876 ms oleg127-client.virtnet: 1 71557 1.45 MB/sec execute 233 sec latency 15.201 ms oleg127-client.virtnet: 1 71875 1.45 MB/sec execute 234 sec latency 21.986 ms oleg127-client.virtnet: 1 72213 1.45 MB/sec execute 235 sec latency 14.899 ms oleg127-client.virtnet: 1 72622 1.46 MB/sec execute 236 sec latency 16.057 ms oleg127-client.virtnet: 1 72982 1.45 MB/sec execute 237 sec latency 16.691 ms oleg127-client.virtnet: 1 73387 1.45 MB/sec execute 238 sec latency 15.279 ms oleg127-client.virtnet: 1 73925 1.47 MB/sec execute 239 sec latency 14.956 ms oleg127-client.virtnet: 1 74571 1.48 MB/sec execute 240 sec latency 15.478 ms oleg127-client.virtnet: 1 74869 1.48 MB/sec execute 241 sec latency 15.064 ms oleg127-client.virtnet: Failover mds1 to oleg127-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 11:53:33 (1713282813) targets are mounted 11:53:33 (1713282813) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 75164 1.48 MB/sec execute 242 sec latency 15.415 ms oleg127-client.virtnet: 1 75475 1.48 MB/sec execute 243 sec latency 23.092 ms oleg127-client.virtnet: 1 75824 1.48 MB/sec execute 244 sec latency 16.027 ms oleg127-client.virtnet: 1 76234 1.48 MB/sec execute 245 sec latency 15.444 ms oleg127-client.virtnet: 1 76597 1.48 MB/sec execute 246 sec latency 17.258 ms oleg127-client.virtnet: 1 76990 1.48 MB/sec execute 247 sec latency 15.189 ms oleg127-client.virtnet: 1 77554 1.49 MB/sec execute 248 sec latency 15.151 ms oleg127-client.virtnet: 1 78183 1.51 MB/sec execute 249 sec latency 16.016 ms oleg127-client.virtnet: 1 78469 1.51 MB/sec execute 250 sec latency 15.857 ms oleg127-client.virtnet: 1 78777 1.50 MB/sec execute 251 sec latency 17.729 ms oleg127-client.virtnet: 1 79076 1.50 MB/sec execute 252 sec latency 16.052 ms oleg127-client.virtnet: 1 79567 1.51 MB/sec execute 253 sec latency 14.823 ms oleg127-client.virtnet: 1 80017 1.51 MB/sec execute 254 sec latency 11.388 ms oleg127-client.virtnet: 1 80506 1.51 MB/sec execute 255 sec latency 11.881 ms oleg127-client.virtnet: 1 81254 1.52 MB/sec execute 256 sec latency 12.921 ms oleg127-client.virtnet: 1 81834 1.54 MB/sec execute 257 sec latency 11.889 ms oleg127-client.virtnet: 1 82159 1.53 MB/sec execute 258 sec latency 13.443 ms oleg127-client.virtnet: 1 82775 1.54 MB/sec execute 259 sec latency 11.233 ms oleg127-client.virtnet: 1 83473 1.54 MB/sec execute 260 sec latency 8.920 ms oleg127-client.virtnet: 1 84507 1.56 MB/sec execute 261 sec latency 7.459 ms oleg127-client.virtnet: 1 85493 1.58 MB/sec execute 262 sec latency 7.415 ms oleg127-client.virtnet: 1 86141 1.58 MB/sec execute 263 sec latency 8.976 ms oleg127-client.virtnet: 1 86953 1.59 MB/sec execute 264 sec latency 8.280 ms oleg127-client.virtnet: 1 87784 1.60 MB/sec execute 265 sec latency 7.153 ms oleg127-client.virtnet: 1 88995 1.63 MB/sec execute 266 sec latency 7.493 ms oleg127-client.virtnet: 1 89638 1.63 MB/sec execute 267 sec latency 9.895 ms oleg127-client.virtnet: 1 90443 1.64 MB/sec execute 268 sec latency 7.595 ms oleg127-client.virtnet: 1 91292 1.64 MB/sec execute 269 sec latency 7.474 ms oleg127-client.virtnet: 1 92503 1.68 MB/sec execute 270 sec latency 7.477 ms oleg127-client.virtnet: 1 93101 1.67 MB/sec execute 271 sec latency 9.322 ms oleg127-client.virtnet: 1 93907 1.69 MB/sec execute 272 sec latency 7.652 ms oleg127-client.virtnet: 1 94713 1.69 MB/sec execute 273 sec latency 8.151 ms oleg127-client.virtnet: 1 95946 1.72 MB/sec execute 274 sec latency 7.583 ms oleg127-client.virtnet: 1 96566 1.72 MB/sec execute 275 sec latency 10.857 ms oleg127-client.virtnet: 1 97346 1.73 MB/sec execute 276 sec latency 7.811 ms oleg127-client.virtnet: 1 98185 1.73 MB/sec execute 277 sec latency 8.559 ms oleg127-client.virtnet: 1 99381 1.76 MB/sec execute 278 sec latency 7.461 ms oleg127-client.virtnet: 1 100026 1.76 MB/sec execute 279 sec latency 8.171 ms oleg127-client.virtnet: 1 100787 1.77 MB/sec execute 280 sec latency 11.414 ms oleg127-client.virtnet: 1 101548 1.77 MB/sec execute 281 sec latency 8.114 ms oleg127-client.virtnet: 1 102714 1.79 MB/sec execute 282 sec latency 7.647 ms oleg127-client.virtnet: 1 103400 1.80 MB/sec execute 283 sec latency 7.352 ms oleg127-client.virtnet: 1 104130 1.81 MB/sec execute 284 sec latency 9.619 ms oleg127-client.virtnet: 1 104933 1.81 MB/sec execute 285 sec latency 8.558 ms oleg127-client.virtnet: 1 105919 1.83 MB/sec execute 286 sec latency 7.547 ms oleg127-client.virtnet: 1 106854 1.85 MB/sec execute 287 sec latency 7.794 ms oleg127-client.virtnet: 1 107535 1.85 MB/sec execute 288 sec latency 9.169 ms oleg127-client.virtnet: 1 108311 1.85 MB/sec execute 289 sec latency 8.036 ms oleg127-client.virtnet: 1 109331 1.87 MB/sec execute 290 sec latency 7.448 ms oleg127-client.virtnet: 1 110309 1.89 MB/sec execute 291 sec latency 7.508 ms oleg127-client.virtnet: 1 110932 1.88 MB/sec execute 292 sec latency 7.863 ms oleg127-client.virtnet: 1 111721 1.89 MB/sec execute 293 sec latency 8.186 ms oleg127-client.virtnet: 1 112565 1.89 MB/sec execute 294 sec latency 7.555 ms oleg127-client.virtnet: 1 113781 1.92 MB/sec execute 295 sec latency 7.609 ms oleg127-client.virtnet: 1 114417 1.92 MB/sec execute 296 sec latency 9.862 ms oleg127-client.virtnet: 1 115202 1.93 MB/sec execute 297 sec latency 7.614 ms oleg127-client.virtnet: 1 116026 1.93 MB/sec execute 298 sec latency 7.878 ms oleg127-client.virtnet: 1 117246 1.96 MB/sec execute 299 sec latency 7.575 ms oleg127-client.virtnet: 1 cleanup 300 sec oleg127-client.virtnet: 0 cleanup 300 sec oleg127-client.virtnet: oleg127-client.virtnet: Operation Count AvgLat MaxLat oleg127-client.virtnet: ---------------------------------------- oleg127-client.virtnet: NTCreateX 18788 9.695 25699.119 oleg127-client.virtnet: Close 13776 1.384 5.733 oleg127-client.virtnet: Rename 796 9.122 20.405 oleg127-client.virtnet: Unlink 3820 2.349 7.073 oleg127-client.virtnet: Qpathinfo 17029 2.878 19930.472 oleg127-client.virtnet: Qfileinfo 2968 0.172 2.414 oleg127-client.virtnet: Qfsinfo 3142 0.289 26.185 oleg127-client.virtnet: Sfileinfo 1522 4.879 14.489 oleg127-client.virtnet: Find 6598 0.492 14.353 oleg127-client.virtnet: WriteX 9286 1.089 7.629 oleg127-client.virtnet: ReadX 29576 0.021 2.707 oleg127-client.virtnet: LockX 62 1.482 2.060 oleg127-client.virtnet: UnlockX 62 1.584 3.696 oleg127-client.virtnet: Flush 1315 7.388 587.108 oleg127-client.virtnet: oleg127-client.virtnet: Throughput 1.95958 MB/sec 1 clients 1 procs max_latency=25699.135 ms oleg127-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg127-client.virtnet at Tue Apr 16 11:54:19 EDT 2024 with return code 0 oleg127-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg127-client.virtnet oleg127-client.virtnet: /mnt/lustre/d70b.replay-single/oleg127-client.virtnet /mnt/lustre/d70b.replay-single/oleg127-client.virtnet oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg127-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg127-client.virtnet: removed directory: 'clients/client0' oleg127-client.virtnet: removed directory: 'clients' oleg127-client.virtnet: removed 'client.txt' oleg127-client.virtnet: /mnt/lustre/d70b.replay-single/oleg127-client.virtnet oleg127-client.virtnet: dbench successfully finished