[ 1928.444267] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1929.478638] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1931.409391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1931.769348] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1932.246539] Lustre: Failing over lustre-MDT0000
[ 1932.283272] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping)
[ 1932.313173] Lustre: server umount lustre-MDT0000 complete
[ 1935.067172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1935.067350] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1935.067351] LustreError: Skipped 1 previous similar message
[ 1935.079961] Lustre: Skipped 4 previous similar messages
[ 1945.242262] LDISKFS-fs (dm-0): recovery complete
[ 1945.243348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1945.271654] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1945.335523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1945.343822] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1946.071301] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1947.930812] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1950.348323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to  (at 0@lo)
[ 1950.350826] Lustre: Skipped 1 previous similar message
[ 1950.356925] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 1950.372139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3663 to 0x2c0000401:3681)
[ 1950.372142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3698 to 0x280000401:3713)
[ 1950.900788] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1951.271310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1953.316652] Lustre: DEBUG MARKER: (dd_pid=18647, time=0, timeout=600)
[ 1959.712042] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1960.772168] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1962.705045] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1963.056047] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1963.539301] Lustre: Failing over lustre-MDT0000
[ 1963.591586] Lustre: server umount lustre-MDT0000 complete
[ 1976.502187] LDISKFS-fs (dm-0): recovery complete
[ 1976.503508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1976.531218] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1976.593807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1977.351115] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1981.615670] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3663 to 0x2c0000401:3713)
[ 1981.615679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3715 to 0x280000401:3745)
[ 1982.119953] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1982.484570] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1984.501022] Lustre: DEBUG MARKER: (dd_pid=20929, time=0, timeout=600)
[ 1995.504165] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 03:55:11 (1713426911)
[ 2001.649489] LustreError: 17791:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 2001.653969] LustreError: 17791:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 2016.475901] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 03:55:32 (1713426932)
[ 2024.151139] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2026.544107] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2026.903347] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2027.250045] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2027.620075] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2029.017145] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2030.429737] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2031.801344] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2033.174111] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2034.561561] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2035.953221] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2037.338096] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2038.721879] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2040.137290] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2041.549099] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2042.937052] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2044.333072] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2045.728595] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2047.154934] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2048.526117] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2049.920732] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2051.313995] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2052.717908] Lustre: DEBUG MARKER: Set quota for 19 times
[ 2054.120714] Lustre: DEBUG MARKER: Set quota for 20 times
[ 2055.513382] Lustre: DEBUG MARKER: Set quota for 21 times
[ 2056.894151] Lustre: DEBUG MARKER: Set quota for 22 times
[ 2071.812556] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 03:56:27 (1713426987)
[ 2081.755487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2081.757756] Lustre: Skipped 3 previous similar messages
[ 2083.268391] Lustre: server umount lustre-MDT0000 complete
[ 2084.209545] LustreError: 6916:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713427001 with bad export cookie 6276795171754348692
[ 2084.210468] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2084.219429] LustreError: 6916:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2084.313187] Lustre: server umount lustre-MDT0001 complete
[ 2095.284924] Lustre: server umount lustre-OST0000 complete
[ 2106.270681] Lustre: server umount lustre-OST0001 complete
[ 2109.250051] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local
[ 2111.383319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2111.503185] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2111.507629] LustreError: Skipped 27 previous similar messages
[ 2111.523552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2112.237038] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2114.332020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2114.447605] mount.lustre (23548) used greatest stack depth: 8944 bytes left
[ 2115.159843] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2117.257523] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2117.260567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2118.556016] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2120.719125] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 2120.721878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2121.765824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3747 to 0x280000401:3777)
[ 2121.777822] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3716 to 0x2c0000401:3745)
[ 2121.981383] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2123.765647] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 2128.180500] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2129.000063] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 2139.916584] Lustre: server umount lustre-MDT0000 complete
[ 2140.867259] LustreError: 22670:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713427058 with bad export cookie 6276795171754355846
[ 2140.868491] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2140.874331] LustreError: 22670:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2143.803267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 2143.805564] Lustre: Skipped 1 previous similar message
[ 2145.807134] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 2145.809719] Lustre: Skipped 1 previous similar message
[ 2146.973396] Lustre: server umount lustre-MDT0001 complete
[ 2147.945044] Lustre: server umount lustre-OST0000 complete
[ 2148.938700] Lustre: server umount lustre-OST0001 complete
[ 2151.971951] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local
[ 2154.146957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2154.291906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2154.293666] Lustre: Skipped 3 previous similar messages
[ 2155.016007] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2157.109440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2157.931056] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2160.038909] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2160.041983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2161.273174] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2163.364630] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 2163.367880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2164.414126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3747 to 0x280000401:3809)
[ 2164.419636] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3716 to 0x2c0000401:3777)
[ 2164.615953] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2167.413733] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 2170.813149] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2171.647066] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 2180.390629] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 03:58:16 (1713427096)
[ 2180.742662] Lustre: DEBUG MARKER: OST0_SIZE: 3605408 required: 6144
[ 2181.096117] Lustre: DEBUG MARKER: run for 4MB test file
[ 2184.727682] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 2185.771531] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2186.114901] Lustre: DEBUG MARKER: Write half of file
[ 2186.526691] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2186.915688] Lustre: DEBUG MARKER: Step1: done
[ 2187.260510] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2187.639466] Lustre: DEBUG MARKER: Step2: done
[ 2197.769380] Lustre: DEBUG MARKER: OST0_SIZE: 3605408 required: 61440
[ 2198.127787] Lustre: DEBUG MARKER: run for 40MB test file
[ 2200.506717] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 2201.527465] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2201.868239] Lustre: DEBUG MARKER: Write half of file
[ 2202.608177] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2203.357611] Lustre: DEBUG MARKER: Step1: done
[ 2203.703713] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2204.077365] Lustre: DEBUG MARKER: Step2: done
[ 2223.762914] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 03:58:59 (1713427139)
[ 2239.355265] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 03:59:15 (1713427155)
[ 2249.441497] Lustre: DEBUG MARKER: Write...
[ 2249.991265] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2277.676046] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 03:59:53 (1713427193)
[ 2280.629918] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 03:59:56 (1713427196)
[ 2285.692478] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:00:01 (1713427201)
[ 2289.859488] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:00:05 (1713427205)
[ 2293.802889] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:00:09 (1713427209)
[ 2318.577018] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2353.655827] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2409.827804] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:02:05 (1713427325)
[ 2424.139520] Lustre: DEBUG MARKER: Restart...
[ 2427.627295] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2427.630960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2427.636851] Lustre: Skipped 12 previous similar messages
[ 2427.638929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2430.824954] Lustre: server umount lustre-MDT0000 complete
[ 2431.743386] LustreError: 547:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713427349 with bad export cookie 6276795171754358114
[ 2431.744583] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2431.748277] LustreError: 547:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2431.842623] Lustre: server umount lustre-MDT0001 complete
[ 2443.113077] Lustre: server umount lustre-OST0000 complete
[ 2454.166892] Lustre: server umount lustre-OST0001 complete
[ 2457.385971] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local
[ 2459.490582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2459.613145] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2459.619176] LustreError: Skipped 5 previous similar messages
[ 2459.636607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2459.638794] Lustre: Skipped 3 previous similar messages
[ 2460.374727] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2462.758380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2463.564928] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2465.814401] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2465.820302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2467.092740] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2469.248699] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 2469.252053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2470.299742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3821 to 0x280000401:3841)
[ 2470.300365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3785 to 0x2c0000401:3809)
[ 2470.610608] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2473.301298] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 2476.993592] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2477.840295] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 2500.736613] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:03:36 (1713427416)
[ 2523.980165] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:04:00 (1713427440)
[ 2963.702658] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:11:19 (1713427879)
[ 2968.651243] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2968.653706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2968.657639] Lustre: Skipped 3 previous similar messages
[ 2968.659723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2968.661958] Lustre: Skipped 3 previous similar messages
[ 2971.346891] Lustre: server umount lustre-MDT0000 complete
[ 2972.340490] LustreError: 32060:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713427889 with bad export cookie 6276795171754366927
[ 2972.341590] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2972.347374] LustreError: 32060:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2972.440138] Lustre: server umount lustre-MDT0001 complete
[ 2983.433276] Lustre: server umount lustre-OST0000 complete
[ 2994.806304] Lustre: server umount lustre-OST0001 complete
[ 2998.977470] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local
[ 3001.674298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3001.808873] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3001.831877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3001.833560] Lustre: Skipped 3 previous similar messages
[ 3002.559807] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3004.806218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3005.635160] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3007.806637] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3007.809721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3009.029426] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3011.228309] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3011.234661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3012.319429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8844 to 0x280000401:8865)
[ 3012.329489] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8810 to 0x2c0000401:8833)
[ 3012.635235] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3015.318222] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 3018.795004] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3019.630364] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3037.569986] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:12:33 (1713427953)
[ 3050.452151] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:12:46 (1713427966)
[ 3065.623975] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:13:01 (1713427981)
[ 3080.184015] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:13:16 (1713427996)
[ 3094.605743] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:13:30 (1713428010)
[ 3097.771498] LustreError: 23820:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3097.776664] LustreError: 23820:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 10 previous similar messages
[ 3121.218530] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:13:57 (1713428037)
[ 3160.464002] Lustre: DEBUG MARKER: == sanity-quota test 49: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:14:36 (1713428076)
[ 3162.165340] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3162.167490] Lustre: Skipped 1 previous similar message
[ 3163.167271] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3163.169634] Lustre: Skipped 163 previous similar messages
[ 3165.175442] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3165.178147] Lustre: Skipped 327 previous similar messages
[ 3169.180647] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3169.181630] Lustre: Skipped 891 previous similar messages
[ 3177.179858] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3177.180941] Lustre: Skipped 1908 previous similar messages
[ 3193.183242] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3193.184766] Lustre: Skipped 3474 previous similar messages
[ 3310.379411] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3310.381846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3310.383364] Lustre: Skipped 3 previous similar messages
[ 3313.395089] Lustre: server umount lustre-MDT0000 complete
[ 3314.827066] LustreError: 12005:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713428232 with bad export cookie 6276795171756049853
[ 3314.829624] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3314.845731] LustreError: 12005:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3320.956096] Lustre: server umount lustre-MDT0001 complete
[ 3322.500589] Lustre: server umount lustre-OST0000 complete
[ 3324.020940] Lustre: server umount lustre-OST0001 complete
[ 3326.039876] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid
[ 3328.025359] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local
[ 3333.672233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3338.265199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3340.664249] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3340.666863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3343.315989] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3343.319083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3347.065386] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local
[ 3350.258991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3350.298637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3350.398654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
[ 3350.407164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 3350.463032] Lustre: lustre-MDT0000: new disk, initializing
[ 3350.488501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3350.492556] Lustre: Skipped 3 previous similar messages
[ 3350.500802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 3350.505332] Lustre: Skipped 1 previous similar message
[ 3351.656126] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3356.031472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3356.062115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3356.102073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 3356.105484] Lustre: Skipped 1 previous similar message
[ 3356.159138] Lustre: lustre-MDT0001: new disk, initializing
[ 3356.188441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 3356.194321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 3357.244268] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3361.305297] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3361.311007] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3361.331224] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3361.337032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3361.422162] Lustre: lustre-OST0000: new disk, initializing
[ 3361.424932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 3362.818136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 3362.822091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 3362.838836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 3363.071686] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3367.502797] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3367.508202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3367.529810] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3367.534922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3367.575310] Lustre: lustre-OST0001: new disk, initializing
[ 3367.578708] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 3369.380058] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3369.454952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 3369.459804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 3369.473832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 3374.003286] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3375.055459] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3375.056942] Lustre: Skipped 1 previous similar message
[ 3390.358035] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:18:26 (1713428306)
[ 3405.010263] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:18:40 (1713428320)
[ 3426.150348] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:19:02 (1713428342)
[ 3431.258907] Lustre: DEBUG MARKER: rename directory return 255
[ 3450.209574] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:19:26 (1713428366)
[ 3458.975266] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:19:34 (1713428374)
[ 3470.514220] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:19:46 (1713428386)
[ 3503.192654] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:20:19 (1713428419)
[ 3513.698327] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:20:29 (1713428429)
[ 3530.760663] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:20:46 (1713428446)
[ 3578.898343] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:21:35 (1713428495)
[ 3579.803377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3579.803770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3579.803771] Lustre: Skipped 8 previous similar messages
[ 3579.809043] Lustre: Skipped 12 previous similar messages
[ 3585.585009] Lustre: server umount lustre-MDT0000 complete
[ 3586.716660] LustreError: 4772:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713428504 with bad export cookie 6276795171756421406
[ 3586.718133] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3586.729391] LustreError: 4772:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 3586.821991] Lustre: server umount lustre-MDT0001 complete
[ 3597.956488] Lustre: server umount lustre-OST0000 complete
[ 3609.396574] Lustre: server umount lustre-OST0001 complete
[ 3615.176014] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local
[ 3617.878390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3618.022600] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3618.026443] LustreError: Skipped 4 previous similar messages
[ 3618.038508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3618.041413] Lustre: Skipped 3 previous similar messages
[ 3618.043653] LustreError: 0-0: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3619.129403] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3621.739074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3621.849636] LustreError: 0-0: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3622.736942] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3625.822833] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3625.826780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3625.949820] LustreError: 0-0: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3627.665105] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3627.948227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 3630.336612] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3630.340357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3630.374778] LustreError: 0-0: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3631.586882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 3632.020711] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3634.810164] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3635.795081] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3642.837264] LustreError: 27952:0:(osd_handler.c:3339:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 3646.875388] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3646.880460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3646.883562] Lustre: Skipped 7 previous similar messages
[ 3649.833914] Lustre: server umount lustre-MDT0000 complete
[ 3651.100074] LustreError: 27942:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713428568 with bad export cookie 6276795171756470798
[ 3651.102116] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3651.112580] LustreError: 27942:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3651.239091] Lustre: server umount lustre-MDT0001 complete
[ 3662.655321] Lustre: server umount lustre-OST0000 complete
[ 3674.113516] Lustre: server umount lustre-OST0001 complete
[ 3680.060901] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local
[ 3683.197091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3684.486312] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3687.607487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3688.717518] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3691.798590] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3691.804196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3692.918191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[ 3693.628068] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3696.771663] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3696.777495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3697.844286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[ 3698.657566] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3701.934494] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3703.225482] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 3723.754664] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:23:59 (1713428639)
[ 3726.844109] LustreError: 7311:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3726.851548] LustreError: 7311:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 3741.654830] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 3743.764303] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:24:19 (1713428659)
[ 3751.250484] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 3753.320669] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 04:24:29 (1713428669)
[ 3768.077112] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 3770.012504] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:24:45 (1713428685)
[ 3788.515088] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:25:04 (1713428704)
[ 3792.069906] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 3793.702145] Lustre: 14637:0:(qsd_reint.c:256:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1005:0x0] is 0, but index isn't empty (1)
[ 3794.831490] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3796.320819] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3797.563386] Lustre: DEBUG MARKER: Write...
[ 3805.735901] LustreError: 16246:0:(mgs_handler.c:866:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3809.904924] Lustre: DEBUG MARKER: Write...
[ 3813.092333] Lustre: DEBUG MARKER: Write...
[ 3854.159497] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:26:10 (1713428770)
[ 3865.107157] LustreError: 21927:0:(mgs_handler.c:866:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3884.515899] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:26:40 (1713428800)
[ 3887.750214] LustreError: 25041:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 3887.756562] LustreError: 25041:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 3895.376350] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 3895.730157] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 3923.369738] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:27:19 (1713428839)
[ 3940.313717] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:27:36 (1713428856)
[ 3958.099165] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:27:54 (1713428874)
[ 3960.393117] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[ 3971.053703] Lustre: DEBUG MARKER: Write...
[ 3971.492750] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4012.706050] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:28:48 (1713428928)
[ 4015.094870] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[ 4052.781805] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:29:28 (1713428968)
[ 4055.060949] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 4059.983363] Lustre: DEBUG MARKER: Write...
[ 4060.361342] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4083.075102] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:29:59 (1713428999)
[ 4091.059104] Lustre: DEBUG MARKER: set to use default quota
[ 4091.382058] Lustre: DEBUG MARKER: set default quota
[ 4091.705944] Lustre: DEBUG MARKER: get default quota
[ 4092.644096] Lustre: DEBUG MARKER: Test not out of quota
[ 4093.518618] Lustre: DEBUG MARKER: Test out of quota
[ 4096.128651] Lustre: DEBUG MARKER: Increase default quota
[ 4104.249202] Lustre: DEBUG MARKER: Set quota to override default quota
[ 4106.787376] Lustre: DEBUG MARKER: Set to use default quota again
[ 4115.502686] Lustre: DEBUG MARKER: Cleanup
[ 4140.026794] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 04:30:56 (1713429056)
[ 4146.792987] Lustre: DEBUG MARKER: set default quota for qpool1
[ 4147.290668] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 4173.493063] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 04:31:29 (1713429089)
[ 4206.767925] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 04:32:02 (1713429122)
[ 4265.899851] Lustre: DEBUG MARKER: Write...
[ 4266.586052] Lustre: DEBUG MARKER: Write out of block quota ...
[ 4338.306068] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 04:34:14 (1713429254)
[ 4353.811139] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 04:34:29 (1713429269)
[ 4357.107535] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 04:34:33 (1713429273)
[ 4373.158383] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 04:34:49 (1713429289)
[ 4388.241003] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 04:35:04 (1713429304)
[ 4398.259252] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 04:35:14 (1713429314)
[ 4409.775121] Lustre: *** cfs_fail_loc=a06, val=0***
[ 4409.777307] Lustre: Skipped 1231 previous similar messages
[ 4409.860792] LustreError: 1950:0:(qmt_lock.c:457:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
  qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:23552 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[ 4413.501984] Lustre: Failing over lustre-OST0001
[ 4413.523218] Lustre: server umount lustre-OST0001 complete
[ 4413.995590] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 4413.996114] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4413.996115] Lustre: Skipped 4 previous similar messages
[ 4413.996306] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4413.996307] LustreError: Skipped 3 previous similar messages
[ 4414.012525] LustreError: Skipped 1 previous similar message
[ 4415.985544] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 4415.988750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 4416.055482] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 4416.061944] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect
[ 4416.064197] Lustre: Skipped 1 previous similar message
[ 4417.306638] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4417.723102] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 4417.726155] Lustre: Skipped 1 previous similar message
[ 4417.847207] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 4417.847246] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.204.152@tcp (at 0@lo)
[ 4417.847247] Lustre: Skipped 7 previous similar messages
[ 4417.856873] Lustre: Skipped 1 previous similar message
[ 4443.902085] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 04:35:59 (1713429359)
[ 4446.469459] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 4450.841602] LustreError: 13166:0:(qmt_pool.c:1302:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 4453.931213] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4453.933670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4453.935675] Lustre: Skipped 4 previous similar messages
[ 4460.911837] LustreError: 13166:0:(qmt_pool.c:1302:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 4460.944167] Lustre: server umount lustre-MDT0000 complete
[ 4463.227246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4463.262053] LustreError: 166-1: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4463.331842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4463.334357] Lustre: Skipped 7 previous similar messages
[ 4463.343064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129)
[ 4463.343066] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129)
[ 4464.126094] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4468.332586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to  (at 0@lo)
[ 4468.334831] Lustre: Skipped 1 previous similar message
[ 4476.385232] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 04:36:32 (1713429392)
[ 4477.788392] LustreError: 23472:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 4477.793123] LustreError: 23472:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 3 previous similar messages
[ 4483.157416] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 04:36:39 (1713429399)
[ 4489.309139] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 04:36:45 (1713429405)
[ 4504.285505] Lustre: *** cfs_fail_loc=a08, val=0***
[ 4504.286705] Lustre: Skipped 8 previous similar messages
[ 4504.288240] Lustre: *** cfs_fail_loc=a08, val=0***
[ 4504.289696] Lustre: Skipped 1 previous similar message
[ 4537.542214] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 04:37:33 (1713429453)
[ 4577.835427] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 04:38:13 (1713429493)
[ 4725.388463] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed