== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:28:25 (1713295705) start mds service on oleg325-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: detected 4 online CPUs by sysfs oleg325-server: Force libcfs to create 2 CPU partitions oleg325-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg325-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg325-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg325-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0000 oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg325-client.virtnet: -o user_xattr,flock oleg325-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 735 (time 1713295758.38 total 10.85 last 67.77) - open/close 1485 (time 1713295773.02 total 25.49 last 51.23) - open/close 2004 (time 1713295783.21 total 35.67 last 50.95) - open/close 2645 (time 1713295793.46 total 45.92 last 62.55) - open/close 3121 (time 1713295803.46 total 55.92 last 47.60) - open/close 3749 (time 1713295814.05 total 66.52 last 59.28) - open/close 4455 (time 1713295824.33 total 76.79 last 68.69) - open/close 5229 (time 1713295834.74 total 87.20 last 74.37) - open/close 6011 (time 1713295845.18 total 97.64 last 74.91) - open/close 6637 (time 1713295855.48 total 107.94 last 60.77) - open/close 7215 (time 1713295865.63 total 118.10 last 56.92) - open/close 7771 (time 1713295875.82 total 128.28 last 54.57) - open/close 8415 (time 1713295886.00 total 138.46 last 63.26) - open/close 9029 (time 1713295896.82 total 149.29 last 56.73) - open/close 9426 (time 1713295906.98 total 159.44 last 39.10) - open/close 10000 (time 1713295914.21 total 166.68 last 79.31) - open/close 10681 (time 1713295924.86 total 177.33 last 63.95) - open/close 11463 (time 1713295935.33 total 187.79 last 74.75) - open/close 11943 (time 1713295945.39 total 197.86 last 47.68) - open/close 12874 (time 1713295956.18 total 208.64 last 86.34) - open/close 13605 (time 1713295966.64 total 219.10 last 69.87) - open/close 14188 (time 1713295976.98 total 229.45 last 56.35) - open/close 14799 (time 1713295987.16 total 239.63 last 60.02) - open/close 16387 (time 1713295997.16 total 249.63 last 158.79) - open/close 20000 (time 1713296003.64 total 256.10 last 557.94) - open/close 25786 (time 1713296013.64 total 266.10 last 578.60) - open/close 30000 (time 1713296020.92 total 273.38 last 579.10) - open/close 35640 (time 1713296030.92 total 283.38 last 563.97) - open/close 40000 (time 1713296038.50 total 290.96 last 575.18) - open/close 45688 (time 1713296048.50 total 300.96 last 568.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 308.03 seconds: 161.13 ops/second - unlinked 0 (time 1713296056 ; total 0 ; last 0) - unlinked 10000 (time 1713296066 ; total 10 ; last 10) - unlinked 20000 (time 1713296076 ; total 20 ; last 10) - unlinked 30000 (time 1713296085 ; total 29 ; last 9) - unlinked 40000 (time 1713296095 ; total 39 ; last 10) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 48 seconds: 1034.000000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg325-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg325-server unloading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2