== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:24:54 (1713533094) start mds service on oleg233-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing load_modules_local oleg233-server: Loading modules from /home/green/git/lustre-release/lustre oleg233-server: detected 4 online CPUs by sysfs oleg233-server: Force libcfs to create 2 CPU partitions oleg233-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg233-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg233-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock oleg233-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 525 (time 1713533133.95 total 10.01 last 52.43) - open/close 935 (time 1713533144.53 total 20.59 last 38.77) - open/close 1447 (time 1713533155.49 total 31.55 last 46.72) - open/close 1971 (time 1713533165.49 total 41.55 last 52.39) - open/close 2387 (time 1713533176.25 total 52.31 last 38.66) - open/close 2885 (time 1713533186.29 total 62.35 last 49.61) - open/close 3363 (time 1713533196.29 total 72.35 last 47.79) - open/close 3852 (time 1713533206.29 total 82.35 last 48.89) - open/close 4317 (time 1713533216.29 total 92.35 last 46.49) - open/close 4792 (time 1713533226.29 total 102.35 last 47.50) - open/close 5136 (time 1713533236.83 total 112.89 last 32.64) - open/close 5384 (time 1713533247.05 total 123.12 last 24.26) - open/close 5822 (time 1713533257.06 total 133.12 last 43.78) - open/close 6250 (time 1713533267.06 total 143.12 last 42.79) - open/close 6657 (time 1713533277.78 total 153.85 last 37.96) - open/close 7072 (time 1713533288.75 total 164.81 last 37.85) - open/close 7366 (time 1713533299.24 total 175.30 last 28.03) - open/close 7752 (time 1713533310.06 total 186.12 last 35.67) - open/close 8128 (time 1713533320.91 total 196.97 last 34.66) - open/close 8514 (time 1713533331.91 total 207.97 last 35.09) - open/close 8869 (time 1713533342.79 total 218.85 last 32.63) - open/close 9214 (time 1713533353.68 total 229.75 last 31.66) - open/close 9388 (time 1713533363.69 total 239.75 last 17.39) - open/close 9598 (time 1713533373.72 total 249.78 last 20.93) - open/close 9883 (time 1713533384.32 total 260.39 last 26.88) - open/close 10000 (time 1713533392.07 total 268.13 last 15.10) - open/close 10308 (time 1713533402.91 total 278.97 last 28.42) - open/close 10578 (time 1713533413.48 total 289.54 last 25.54) - open/close 10848 (time 1713533424.06 total 300.12 last 25.53) - open/close 11201 (time 1713533434.26 total 310.32 last 34.61) - open/close 12492 (time 1713533444.26 total 320.32 last 129.07) - open/close 14082 (time 1713533454.26 total 330.32 last 158.97) - open/close 15573 (time 1713533464.27 total 340.33 last 149.03) - open/close 17001 (time 1713533474.27 total 350.33 last 142.72) - open/close 18406 (time 1713533484.27 total 360.33 last 140.48) - open/close 19693 (time 1713533494.28 total 370.34 last 128.63) - open/close 20000 (time 1713533496.57 total 372.63 last 134.04) - open/close 21371 (time 1713533506.57 total 382.64 last 137.04) - open/close 22557 (time 1713533516.58 total 392.64 last 118.56) - open/close 23753 (time 1713533526.58 total 402.65 last 119.51) - open/close 24954 (time 1713533536.59 total 412.65 last 120.08) - open/close 26241 (time 1713533546.59 total 422.66 last 128.60) - open/close 27422 (time 1713533556.60 total 432.66 last 118.06) - open/close 28503 (time 1713533566.60 total 442.67 last 108.03) - open/close 29563 (time 1713533576.61 total 452.67 last 105.92) - open/close 30000 (time 1713533580.26 total 456.32 last 119.79) - open/close 31024 (time 1713533590.26 total 466.32 last 102.37) - open/close 32036 (time 1713533600.27 total 476.33 last 101.13) - open/close 33089 (time 1713533610.27 total 486.34 last 105.27) - open/close 34025 (time 1713533620.28 total 496.34 last 93.57) - open/close 35250 (time 1713533630.28 total 506.34 last 122.48) - open/close 36242 (time 1713533640.28 total 516.34 last 99.16) - open/close 37138 (time 1713533650.28 total 526.34 last 89.59) - open/close 37932 (time 1713533660.29 total 536.35 last 79.32) - open/close 38715 (time 1713533670.29 total 546.36 last 78.28) - open/close 39480 (time 1713533680.30 total 556.36 last 76.48) - open/close 40000 (time 1713533687.45 total 563.51 last 72.70) - open/close 40694 (time 1713533697.46 total 573.52 last 69.31) - open/close 41450 (time 1713533707.47 total 583.53 last 75.52) - open/close 42199 (time 1713533717.48 total 593.55 last 74.82) - open/close 42855 (time 1713533727.50 total 603.56 last 65.53) - open/close 43542 (time 1713533737.50 total 613.57 last 68.64) - open/close 44170 (time 1713533747.51 total 623.57 last 62.79) - open/close 44792 (time 1713533757.51 total 633.57 last 62.16) - open/close 45402 (time 1713533767.52 total 643.59 last 60.93) - open/close 46137 (time 1713533777.53 total 653.59 last 73.43) - open/close 46793 (time 1713533787.54 total 663.60 last 65.54) - open/close 47341 (time 1713533797.56 total 673.62 last 54.72) - open/close 47890 (time 1713533807.57 total 683.63 last 54.84) - open/close 48437 (time 1713533817.57 total 693.63 last 54.67) - open/close 48972 (time 1713533827.58 total 703.64 last 53.45) - open/close 49536 (time 1713533837.60 total 713.66 last 56.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 715.64 seconds: 69.35 ops/second - unlinked 0 (time 1713533840 ; total 0 ; last 0) - unlinked 10000 (time 1713533986 ; total 146 ; last 146) - unlinked 20000 (time 1713534093 ; total 253 ; last 107) - unlinked 30000 (time 1713534161 ; total 321 ; last 68) - unlinked 40000 (time 1713534213 ; total 373 ; last 52) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 403 seconds: 123.156326 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg233-server unloading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2