-----============= acceptance-small: sanity-flr ============----- Tue Apr 16 14:58:18 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:58:22 (1713293902) === oleg429-client.virtnet: executing check_config_client /mnt/lustre oleg429-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg429-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800abe83000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800abe83000.idle_timeout=debug disable quota as required oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:58:29 (1713293909) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6917] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 14:58:30 (1713293910) PASS 0a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:58:36 (1713293916) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:58:39 (1713293919) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 14:58:42 (1713293922) PASS 0d (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:58:51 (1713293931) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:58:53 (1713293933) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 14:58:56 (1713293936) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800abe83000.stats=clear osc.lustre-OST0001-osc-ffff8800abe83000.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:59:00 (1713293940) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:59:05 (1713293945) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:59:09 (1713293949) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 14:59:12 (1713293952) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 14:59:17 (1713293957) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:59:21 (1713293961) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:59:24 (1713293964) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:59:28 (1713293968) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 14:59:33 (1713293973) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:59:37 (1713293977) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.915262 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0663698 s, 15.8 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:59:42 (1713293982) 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.143948 s, 146 MB/s PASS 22 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:59:47 (1713293987) Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800abe83000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abe83000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800abe83000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abe83000.ost_server_uuid in DISCONN state after 1 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:00:15 (1713294015) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.151438 s, 48.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800abe83000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abe83000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:00:30 (1713294030) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800abe83000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abe83000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800abe83000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abe83000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:00:55 (1713294055) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.805857 s, 65.1 MB/s -rw-r--r-- 1 root root 52428800 Apr 16 15:00 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.273s user 0m0.000s sys 0m0.115s Stopping /mnt/lustre-ost1 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800abe83000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abe83000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.255s user 0m0.000s sys 0m0.122s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg429-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg429-server oleg429-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800abe83000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abe83000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.316s user 0m0.000s sys 0m0.151s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg429-server: oleg429-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg429-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800abe83000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800abe83000.ost_server_uuid in FULL state after 0 sec PASS 33b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:01:21 (1713294081) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:01:24 (1713294084) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:01:27 (1713294087) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 15:01:29 (1713294089) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 15:01:33 (1713294093) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.207938 s, 60.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:01:39 (1713294099) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010611 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100833 s, 104 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:01:43 (1713294103) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:01:48 (1713294108) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00168486 s, 608 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0010517 s, 974 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00101428 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000604409 s, 1.7 MB/s flv=10 olv=10 PASS 36d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 15:01:53 (1713294113) create test files with size 15729033 .. fd923f825adeacd99ff340bc6dc6cee6 - d1f373bacd921c2a924abe275b7c86a9 - 3d9fbb90cbedf4b963e1e3ad1e5c7814 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre PASS 37 (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 15:02:03 (1713294123) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:02:10 (1713294130) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:02:14 (1713294134) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.031606 s, 33.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00942453 s, 111 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 15:02:18 (1713294138) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0295786 s, 70.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0987457 s, 42.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0530639 s, 39.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 15:02:22 (1713294142) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 15:02:25 (1713294145) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:02:27 (1713294147) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00867566 s, 121 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 15:02:30 (1713294150) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:02:33 (1713294153) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:02:37 (1713294157) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0769336 s, 136 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:02:41 (1713294161) llite.lustre-ffff88012a687000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0845701 s, 124 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg429-client.virtnet /mnt/lustre (opts:) Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a516c800.xattr_cache=1 PASS 44e (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:02:46 (1713294166) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (9s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 15:02:57 (1713294177) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 15:03:05 (1713294185) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 15:03:07 (1713294187) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0637576 s, 49.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0522861 s, 60.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 681812c5d6277e89368ca7afd99a243d - mirror 1 checksum: 86726bcd5c597895d626d01b4ec03af9 - mirror 2 checksum: 681812c5d6277e89368ca7afd99a243d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 15:03:07 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 15:03:07 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 681812c5d6277e89368ca7afd99a243d - mirror 1 checksum: 86726bcd5c597895d626d01b4ec03af9 - mirror 2 checksum: 86726bcd5c597895d626d01b4ec03af9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 15:03:07 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 15:03:07 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:03:11 (1713294191) Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 15:03:13 (1713294193) shut down Failover mds1 to oleg429-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 15:03:25 (1713294205) targets are mounted 15:03:25 (1713294205) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899439 s, 117 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:03:31 (1713294211) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 15:03:34 (1713294214) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:03:36 (1713294216) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00210145 s, 7.8 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 15:03:40 (1713294220) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:03:42 (1713294222) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:03:45 (1713294225) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:03:51 (1713294231) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:04:11 (1713294251) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:04:30 (1713294270) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:04:49 (1713294289) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 15:04:52 (1713294292) mirror create pid 26640 mirror split pid 26641 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26640 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26641 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 15:05:56 (1713294356) Seed set to 4356 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1713294357.065017 write 0x00e1e3 thru 0x01d293 (0xf0b1 bytes) 000002 1713294357.071426 read 0x014d79 thru 0x01d293 (0x851b bytes) 000007 1713294357.073436 mapread 0x008f83 thru 0x00e693 (0x5711 bytes) 000008 1713294357.080146 write 0x02d7d6 thru 0x03ce1c (0xf647 bytes) 000015 1713294357.087546 write 0x02679a thru 0x031d1e (0xb585 bytes) 000016 1713294357.094528 MIRROR_EXTEND to 2 mirrors 000017 1713294357.276973 write 0x008341 thru 0x00b2e2 (0x2fa2 bytes) 000021 1713294357.315797 mapwrite 0x0231b7 thru 0x0291f4 (0x603e bytes) 000024 1713294357.355910 mapread 0x02cf3b thru 0x035a8a (0x8b50 bytes) 000026 1713294357.363398 mapread 0x027070 thru 0x034a24 (0xd9b5 bytes) 000027 1713294357.363944 read 0x01e51b thru 0x0249f9 (0x64df bytes) 000029 1713294357.364631 write 0x0047c6 thru 0x007396 (0x2bd1 bytes) 000030 1713294357.370460 MIRROR_RESYNC 2 mirrors 000031 1713294357.503431 mapwrite 0x03db16 thru 0x03ffff (0x24ea bytes) 000032 1713294357.552663 mapread 0x00e6ac thru 0x00f734 (0x1089 bytes) 000034 1713294357.557114 mapread 0x013eb6 thru 0x0199a8 (0x5af3 bytes) truncating to largest ever: 0x27b78 000036 1713294357.561182 trunc from 0x040000 to 0x027b77 (0x18488 bytes) 000037 1713294357.569199 mapwrite 0x03d394 thru 0x03ffff (0x2c6c bytes) 000039 1713294357.603387 read 0x0251d7 thru 0x0328b9 (0xd6e3 bytes) 000040 1713294357.607037 mapread 0x00d04e thru 0x0136da (0x668d bytes) 000042 1713294357.607853 read 0x017f26 thru 0x024ec7 (0xcfa2 bytes) 000043 1713294357.612716 MIRROR_EXTEND to 3 mirrors 000045 1713294357.798742 MIRROR_RESYNC 3 mirrors 000046 1713294357.954653 trunc from 0x040000 to 0x01db36 (0x224c9 bytes) 000047 1713294357.984872 mapwrite 0x02350d thru 0x02d7c1 (0xa2b5 bytes) 000054 1713294358.032469 mapwrite 0x00eaa8 thru 0x01d216 (0xe76f bytes) 000055 1713294358.060883 mapread 0x01d4cd thru 0x027fff (0xab33 bytes) truncating to largest ever: 0x3d6e0 000057 1713294358.061855 trunc from 0x02d7c2 to 0x03d6df (0xff1e bytes) 000059 1713294358.068190 write 0x014208 thru 0x022a9d (0xe896 bytes) 000063 1713294358.072615 MIRROR_RESYNC 3 mirrors 000064 1713294358.299375 trunc from 0x03d6e0 to 0x03a297 (0x3448 bytes) 000065 1713294358.330593 trunc from 0x03a298 to 0x001e1f (0x38478 bytes) 000066 1713294358.336846 trunc from 0x001e20 to 0x0358ff (0x33ae0 bytes) 000068 1713294358.342619 read 0x016675 thru 0x0202e1 (0x9c6d bytes) 000070 1713294358.347580 trunc from 0x035900 to 0x00b07e (0x2a881 bytes) 000076 1713294358.354256 mapwrite 0x01833e thru 0x019480 (0x1143 bytes) 000077 1713294358.398877 mapwrite 0x02ec84 thru 0x02fc0b (0xf88 bytes) 000078 1713294358.439535 write 0x031d34 thru 0x037257 (0x5524 bytes) 000081 1713294358.441056 mapread 0x003bed thru 0x00fd74 (0xc188 bytes) 000083 1713294358.452052 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000086 1713294358.538244 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000087 1713294358.596405 mapread 0x032a5e thru 0x037257 (0x47fa bytes) 000088 1713294358.607127 trunc from 0x037258 to 0x0384f0 (0x1299 bytes) 000089 1713294358.614657 mapread 0x0127c2 thru 0x016bbe (0x43fd bytes) 000090 1713294358.624978 MIRROR_EXTEND to 2 mirrors 000091 1713294358.849569 mapread 0x016cf9 thru 0x01dee0 (0x71e8 bytes) 000093 1713294358.863211 MIRROR_EXTEND to 3 mirrors 000094 1713294359.081544 read 0x0381a9 thru 0x03832c (0x184 bytes) 000096 1713294359.085065 trunc from 0x0384f1 to 0x00e9b9 (0x29b37 bytes) 000097 1713294359.128971 mapwrite 0x03b386 thru 0x03ffff (0x4c7a bytes) 000098 1713294359.173824 mapread 0x02f41c thru 0x037f7f (0x8b64 bytes) 000099 1713294359.179399 read 0x03e71d thru 0x03ffff (0x18e3 bytes) 000100 1713294359.179880 write 0x02bdb9 thru 0x0376b5 (0xb8fd bytes) 000101 1713294359.185106 trunc from 0x040000 to 0x004586 (0x3ba79 bytes) 000105 1713294359.192932 trunc from 0x004587 to 0x02f3d6 (0x2ae50 bytes) 000108 1713294359.199683 mapwrite 0x01ce89 thru 0x02c63f (0xf7b7 bytes) 000109 1713294359.276028 trunc from 0x02f3d7 to 0x009e3b (0x2559b bytes) 000110 1713294359.283298 write 0x01e8c7 thru 0x022660 (0x3d9a bytes) 000113 1713294359.285077 write 0x0375c3 thru 0x03ffff (0x8a3d bytes) 000114 1713294359.286002 trunc from 0x040000 to 0x011a55 (0x2e5aa bytes) 000117 1713294359.293262 mapwrite 0x03ee2f thru 0x03ffff (0x11d1 bytes) 000119 1713294359.331565 trunc from 0x040000 to 0x00841c (0x37be3 bytes) 000120 1713294359.338655 read 0x0064c8 thru 0x00841c (0x1f55 bytes) 000130 1713294359.343835 mapwrite 0x03b970 thru 0x03ffff (0x4690 bytes) 000133 1713294359.385784 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000134 1713294359.458117 write 0x001ee9 thru 0x004082 (0x219a bytes) 000135 1713294359.464264 MIRROR_EXTEND to 3 mirrors 000136 1713294359.655230 mapwrite 0x02baa4 thru 0x02fd51 (0x42ae bytes) 000137 1713294359.734930 mapread 0x02631f thru 0x032b02 (0xc7e4 bytes) 000138 1713294359.736235 mapwrite 0x02fdaa thru 0x03c6d6 (0xc92d bytes) 000139 1713294359.762988 write 0x029a45 thru 0x035dec (0xc3a8 bytes) 000140 1713294359.763817 mapwrite 0x030fd3 thru 0x034a22 (0x3a50 bytes) 000141 1713294359.784152 trunc from 0x040000 to 0x033f3a (0xc0c5 bytes) 000143 1713294359.799327 read 0x0082b1 thru 0x014ddb (0xcb2b bytes) 000144 1713294359.802612 mapread 0x019ae9 thru 0x020479 (0x6991 bytes) 000146 1713294359.806681 write 0x036b9e thru 0x03ffff (0x9462 bytes) 000148 1713294359.808153 read 0x0327ea thru 0x03c45f (0x9c76 bytes) 000149 1713294359.812821 trunc from 0x040000 to 0x00958f (0x36a70 bytes) 000150 1713294359.819179 write 0x008c99 thru 0x00cfc2 (0x432a bytes) 000151 1713294359.819996 trunc from 0x00cfc3 to 0x001b0d (0xb4b5 bytes) 000154 1713294359.829822 MIRROR_EXTEND to 4 mirrors 000155 1713294360.019573 mapwrite 0x009dd6 thru 0x017f33 (0xe15e bytes) 000156 1713294360.100708 MIRROR_RESYNC 4 mirrors 000157 1713294360.335598 mapread 0x00da09 thru 0x017f33 (0xa52b bytes) 000159 1713294360.347203 MIRROR_EXTEND to 5 mirrors 000160 1713294360.561160 mapwrite 0x03f973 thru 0x03ffff (0x68d bytes) 000161 1713294360.646844 MIRROR_RESYNC 5 mirrors 000162 1713294360.888103 read 0x00ff35 thru 0x01ca9b (0xcb67 bytes) 000164 1713294360.891467 mapread 0x03af5c thru 0x03ffff (0x50a4 bytes) 000168 1713294360.898882 mapread 0x006d31 thru 0x0154f9 (0xe7c9 bytes) 000170 1713294360.906140 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000171 1713294360.987735 read 0x020873 thru 0x02cee0 (0xc66e bytes) 000174 1713294360.993324 trunc from 0x040000 to 0x00baa9 (0x34556 bytes) 000177 1713294361.034674 read 0x00ba5f thru 0x00baa9 (0x4b bytes) 000178 1713294361.039536 write 0x018562 thru 0x018945 (0x3e4 bytes) 000179 1713294361.040172 write 0x03b0e2 thru 0x03e4ad (0x33cc bytes) 000180 1713294361.041177 mapread 0x01502b thru 0x018d4a (0x3d20 bytes) 000182 1713294361.045805 read 0x00fb7b thru 0x01aa8d (0xaf13 bytes) 000183 1713294361.046246 mapwrite 0x0118b2 thru 0x01fd4b (0xe49a bytes) 000184 1713294361.079979 mapwrite 0x034b22 thru 0x03ffff (0xb4de bytes) 000185 1713294361.110715 write 0x007410 thru 0x00a287 (0x2e78 bytes) 000186 1713294361.111378 read 0x0226a9 thru 0x027081 (0x49d9 bytes) 000190 1713294361.112553 read 0x0363ee thru 0x03ffff (0x9c12 bytes) 000192 1713294361.113139 trunc from 0x040000 to 0x01a4ba (0x25b45 bytes) 000193 1713294361.119209 trunc from 0x01a4bb to 0x00e8d7 (0xbbe3 bytes) 000194 1713294361.125818 MIRROR_EXTEND to 5 mirrors 000197 1713294361.279842 mapread 0x00b7d9 thru 0x00e8d7 (0x30ff bytes) 000198 1713294361.293755 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000199 1713294361.378145 read 0x007281 thru 0x00e8d7 (0x7657 bytes) 000200 1713294361.384425 trunc from 0x00e8d8 to 0x00eeca (0x5f3 bytes) 000201 1713294361.416309 mapread 0x005154 thru 0x00b996 (0x6843 bytes) 000202 1713294361.422817 read 0x0058e3 thru 0x00a392 (0x4ab0 bytes) 000203 1713294361.423321 write 0x0367ab thru 0x03c478 (0x5cce bytes) 000204 1713294361.424246 mapread 0x038991 thru 0x03c478 (0x3ae8 bytes) 000205 1713294361.427023 MIRROR_RESYNC 4 mirrors 000208 1713294361.643580 mapread 0x030587 thru 0x03c0aa (0xbb24 bytes) 000209 1713294361.652473 mapread 0x0059f9 thru 0x00bff1 (0x65f9 bytes) 000211 1713294361.656000 read 0x012fbe thru 0x017782 (0x47c5 bytes) 000212 1713294361.658574 trunc from 0x03c479 to 0x00f11f (0x2d359 bytes) 000214 1713294361.688018 write 0x010bee thru 0x01a001 (0x9414 bytes) 000216 1713294361.690882 MIRROR_EXTEND to 5 mirrors 000217 1713294361.848012 trunc from 0x01a002 to 0x004b99 (0x15468 bytes) 000219 1713294361.884729 MIRROR_RESYNC 5 mirrors 000220 1713294362.101599 mapread 0x0036ea thru 0x004b99 (0x14b0 bytes) 000221 1713294362.104092 trunc from 0x004b9a to 0x026213 (0x2167a bytes) 000223 1713294362.129661 MIRROR_RESYNC 5 mirrors 000225 1713294362.381573 trunc from 0x026214 to 0x01193f (0x148d4 bytes) 000226 1713294362.410076 mapread 0x003dac thru 0x009cb2 (0x5f07 bytes) 000227 1713294362.416590 trunc from 0x011940 to 0x02ac0d (0x192ce bytes) 000231 1713294362.424294 trunc from 0x02ac0e to 0x028a12 (0x21fb bytes) 000232 1713294362.429479 write 0x03e8f4 thru 0x03ffff (0x170c bytes) 000233 1713294362.431770 MIRROR_EXTEND to 6 mirrors 000234 1713294362.637273 read 0x02d703 thru 0x02dbcb (0x4c9 bytes) 000235 1713294362.640860 mapwrite 0x0244a9 thru 0x028e3b (0x4993 bytes) 000238 1713294362.716630 read 0x024143 thru 0x027dba (0x3c78 bytes) 000239 1713294362.717430 trunc from 0x040000 to 0x0103c1 (0x2fc3e bytes) 000241 1713294362.727120 read 0x006c55 thru 0x00f35c (0x8708 bytes) 000246 1713294362.732196 write 0x03fd63 thru 0x03ffff (0x29d bytes) 000249 1713294362.733288 read 0x016ace thru 0x024f44 (0xe477 bytes) 000250 1713294362.737646 trunc from 0x040000 to 0x03bf36 (0x40c9 bytes) 000254 1713294362.748469 MIRROR_RESYNC 6 mirrors 000255 1713294363.037939 read 0x00ec66 thru 0x017d12 (0x90ad bytes) 000257 1713294363.043549 MIRROR_EXTEND to 7 mirrors 000258 1713294363.244507 read 0x039767 thru 0x03bf36 (0x27d0 bytes) 000260 1713294363.247437 mapwrite 0x038b2e thru 0x03ffff (0x74d2 bytes) 000261 1713294363.308988 mapwrite 0x016069 thru 0x024c14 (0xebac bytes) 000262 1713294363.333073 mapread 0x020e91 thru 0x0267b3 (0x5923 bytes) 000263 1713294363.335569 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000264 1713294363.416674 read 0x03ecdc thru 0x03ffff (0x1324 bytes) 000265 1713294363.419337 write 0x0085f2 thru 0x00e2aa (0x5cb9 bytes) 000266 1713294363.422761 mapwrite 0x025ae0 thru 0x0277dc (0x1cfd bytes) 000267 1713294363.450934 write 0x00dd1c thru 0x013f72 (0x6257 bytes) 000269 1713294363.454393 mapwrite 0x00dd77 thru 0x01d0b8 (0xf342 bytes) 000270 1713294363.481677 trunc from 0x040000 to 0x013858 (0x2c7a7 bytes) 000272 1713294363.488263 read 0x008e06 thru 0x010538 (0x7733 bytes) truncating to largest ever: 0x3d92e 000274 1713294363.489041 trunc from 0x013859 to 0x03d92d (0x2a0d5 bytes) 000275 1713294363.495144 mapread 0x013bfd thru 0x01da7c (0x9e80 bytes) 000276 1713294363.500275 mapread 0x036817 thru 0x03d92d (0x7117 bytes) 000282 1713294363.502226 trunc from 0x03d92e to 0x01233e (0x2b5ef bytes) 000283 1713294363.509702 write 0x00b7db thru 0x0168b3 (0xb0d9 bytes) 000285 1713294363.510962 read 0x010184 thru 0x0168b3 (0x6730 bytes) 000286 1713294363.511331 trunc from 0x0168b4 to 0x00d341 (0x9572 bytes) 000287 1713294363.520458 MIRROR_EXTEND to 7 mirrors 000290 1713294363.681028 trunc from 0x00d342 to 0x00a7da (0x2b67 bytes) 000291 1713294363.714264 mapwrite 0x03a8e2 thru 0x03ffff (0x571e bytes) 000292 1713294363.753431 read 0x005ab8 thru 0x009fae (0x44f7 bytes) 000298 1713294363.758427 mapread 0x02ca02 thru 0x035e69 (0x9468 bytes) 000299 1713294363.765963 mapwrite 0x017e09 thru 0x01a9a3 (0x2b9b bytes) 000300 1713294363.794210 mapwrite 0x01feec thru 0x02a350 (0xa465 bytes) 000304 1713294363.826489 read 0x037488 thru 0x037b34 (0x6ad bytes) 000305 1713294363.828810 MIRROR_EXTEND to 8 mirrors 000309 1713294363.968949 mapwrite 0x02a667 thru 0x0350c7 (0xaa61 bytes) 000310 1713294364.034113 read 0x0009fb thru 0x0017b1 (0xdb7 bytes) 000312 1713294364.037282 trunc from 0x040000 to 0x027705 (0x188fa bytes) 000314 1713294364.052484 mapread 0x01d517 thru 0x02712c (0x9c16 bytes) 000315 1713294364.056047 trunc from 0x027706 to 0x03389b (0xc196 bytes) 000319 1713294364.062451 mapread 0x017af0 thru 0x025e01 (0xe312 bytes) 000320 1713294364.065953 mapwrite 0x02c156 thru 0x032acd (0x6978 bytes) 000321 1713294364.092549 trunc from 0x03389c to 0x0167ef (0x1d0ac bytes) 000322 1713294364.097104 read 0x01588a thru 0x0167ef (0xf66 bytes) 000323 1713294364.097495 mapwrite 0x019ae0 thru 0x01f2e2 (0x5803 bytes) 000325 1713294364.122099 mapwrite 0x033b24 thru 0x03b9f3 (0x7ed0 bytes) 000326 1713294364.148621 read 0x01bde4 thru 0x028bc4 (0xcde1 bytes) 000327 1713294364.149103 mapwrite 0x01a8c1 thru 0x026dac (0xc4ec bytes) 000328 1713294364.169348 read 0x00c5e6 thru 0x017c5d (0xb678 bytes) 000329 1713294364.169796 write 0x039e3a thru 0x03e2a4 (0x446b bytes) 000331 1713294364.170755 read 0x028ca1 thru 0x0327c0 (0x9b20 bytes) 000332 1713294364.171211 write 0x00b77d thru 0x015f45 (0xa7c9 bytes) 000333 1713294364.171976 mapwrite 0x005c7e thru 0x006825 (0xba8 bytes) 000334 1713294364.189724 write 0x030ae8 thru 0x03435a (0x3873 bytes) 000338 1713294364.192634 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000339 1713294364.249838 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000341 1713294364.308272 trunc from 0x03e2a5 to 0x001334 (0x3cf70 bytes) 000342 1713294364.314259 MIRROR_EXTEND to 7 mirrors 000343 1713294364.451428 trunc from 0x001335 to 0x0206df (0x1f3ab bytes) 000345 1713294364.479190 write 0x01736d thru 0x020bc0 (0x9854 bytes) 000346 1713294364.482995 write 0x03b456 thru 0x03ffff (0x4baa bytes) 000349 1713294364.486403 MIRROR_EXTEND to 8 mirrors 000351 1713294364.628806 read 0x00eb77 thru 0x00f02a (0x4b4 bytes) 000354 1713294364.635098 MIRROR_RESYNC 8 mirrors 000356 1713294364.877352 read 0x012c03 thru 0x01b852 (0x8c50 bytes) 000357 1713294364.880485 write 0x00e38b thru 0x014b06 (0x677c bytes) 000359 1713294364.900333 write 0x00914c thru 0x00d111 (0x3fc6 bytes) 000362 1713294364.904427 write 0x008884 thru 0x00c894 (0x4011 bytes) 000365 1713294364.905776 read 0x03faef thru 0x03ffff (0x511 bytes) 000366 1713294364.909491 write 0x00dddf thru 0x00faeb (0x1d0d bytes) 000367 1713294364.909968 mapwrite 0x03f3f8 thru 0x03ffff (0xc08 bytes) 000368 1713294364.932133 read 0x0365ae thru 0x03f03f (0x8a92 bytes) 000369 1713294364.932591 read 0x01882a thru 0x0236e4 (0xaebb bytes) 000371 1713294364.933403 mapread 0x01bcdf thru 0x01cf12 (0x1234 bytes) 000372 1713294364.933974 mapread 0x0390f1 thru 0x03ffff (0x6f0f bytes) 000376 1713294364.935461 read 0x0371a3 thru 0x03ffff (0x8e5d bytes) 000377 1713294364.935867 read 0x037339 thru 0x03ffff (0x8cc7 bytes) 000378 1713294364.936272 mapread 0x021ded thru 0x02a4f0 (0x8704 bytes) 000381 1713294364.939490 MIRROR_EXTEND to 9 mirrors 000382 1713294365.091329 mapread 0x009cce thru 0x0165d4 (0xc907 bytes) 000383 1713294365.103723 mapwrite 0x020c11 thru 0x02511a (0x450a bytes) 000384 1713294365.162171 mapread 0x0289f8 thru 0x02d7ce (0x4dd7 bytes) 000388 1713294365.166495 read 0x025bb7 thru 0x0320a8 (0xc4f2 bytes) 000389 1713294365.166946 trunc from 0x040000 to 0x0005f3 (0x3fa0c bytes) 000393 1713294365.174781 mapread 0x0000cf thru 0x0005f3 (0x525 bytes) 000399 1713294365.180404 MIRROR_EXTEND to 10 mirrors 000402 1713294365.316713 trunc from 0x0005f4 to 0x03b31d (0x3ad2a bytes) 000403 1713294365.338833 mapread 0x034bd1 thru 0x03b31d (0x674d bytes) 000405 1713294365.345785 read 0x0145df thru 0x0243b6 (0xfdd8 bytes) 000406 1713294365.348852 write 0x007940 thru 0x00dfd5 (0x6696 bytes) 000408 1713294365.352371 write 0x013e9f thru 0x01472d (0x88f bytes) 000410 1713294365.356248 MIRROR_RESYNC 10 mirrors 000412 1713294365.731073 mapwrite 0x0362b5 thru 0x03ffff (0x9d4b bytes) 000414 1713294365.780635 mapread 0x02de05 thru 0x031d5b (0x3f57 bytes) 000417 1713294365.785711 MIRROR_RESYNC 10 mirrors 000418 1713294366.057368 trunc from 0x040000 to 0x007a60 (0x3859f bytes) 000422 1713294366.082944 mapwrite 0x0077b0 thru 0x017326 (0xfb77 bytes) 000423 1713294366.118935 mapread 0x00b6cc thru 0x0172b3 (0xbbe8 bytes) 000424 1713294366.119508 mapwrite 0x0148fd thru 0x01efd1 (0xa6d5 bytes) 000426 1713294366.143195 read 0x002f01 thru 0x00df63 (0xb063 bytes) 000429 1713294366.143901 mapwrite 0x00742e thru 0x00933a (0x1f0d bytes) 000431 1713294366.156844 mapread 0x00bad3 thru 0x00e326 (0x2854 bytes) 000432 1713294366.157335 read 0x01068f thru 0x014bb8 (0x452a bytes) 000435 1713294366.159369 MIRROR_EXTEND to 11 mirrors 000436 1713294366.304973 read 0x001fbe thru 0x00a5a9 (0x85ec bytes) 000437 1713294366.310988 read 0x01dd39 thru 0x01efd1 (0x1299 bytes) 000441 1713294366.325020 read 0x013a0e thru 0x01efd1 (0xb5c4 bytes) 000442 1713294366.328026 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000443 1713294366.408833 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000445 1713294366.501486 mapread 0x000f37 thru 0x00f8e0 (0xe9aa bytes) 000449 1713294366.531840 write 0x00e0c1 thru 0x011687 (0x35c7 bytes) 000451 1713294366.556494 trunc from 0x01efd2 to 0x021a81 (0x2ab0 bytes) 000452 1713294366.563661 write 0x030f09 thru 0x035d80 (0x4e78 bytes) 000453 1713294366.564667 write 0x03b8b8 thru 0x03f3eb (0x3b34 bytes) 000454 1713294366.565303 write 0x021aec thru 0x02e9ec (0xcf01 bytes) 000455 1713294366.570501 mapread 0x00bfa4 thru 0x00ca80 (0xadd bytes) 000456 1713294366.570991 write 0x0206a7 thru 0x029575 (0x8ecf bytes) 000459 1713294366.572186 write 0x00f8c5 thru 0x01da66 (0xe1a2 bytes) 000460 1713294366.572976 mapread 0x03ac0a thru 0x03f3eb (0x47e2 bytes) 000461 1713294366.576256 write 0x02b8e6 thru 0x02f024 (0x373f bytes) 000462 1713294366.578376 trunc from 0x03f3ec to 0x03362a (0xbdc1 bytes) 000463 1713294366.582751 trunc from 0x03362b to 0x02afa7 (0x8683 bytes) 000464 1713294366.586398 mapread 0x029ea0 thru 0x02afa7 (0x1108 bytes) 000467 1713294366.587639 mapread 0x009386 thru 0x0155be (0xc239 bytes) 000468 1713294366.588102 write 0x03270e thru 0x039100 (0x69f3 bytes) 000471 1713294366.591041 MIRROR_EXTEND to 10 mirrors 000473 1713294366.762921 mapread 0x00ddad thru 0x015fd9 (0x822d bytes) 000475 1713294366.780303 trunc from 0x039101 to 0x017d6a (0x21396 bytes) 000477 1713294366.804200 mapread 0x005353 thru 0x0071bd (0x1e6b bytes) 000482 1713294366.808724 mapread 0x003a48 thru 0x012c46 (0xf1ff bytes) 000483 1713294366.812062 read 0x001aa8 thru 0x002ccb (0x1224 bytes) 000486 1713294366.813187 mapread 0x006828 thru 0x009e29 (0x3602 bytes) 000488 1713294366.814022 mapread 0x00fe7d thru 0x017d6a (0x7eee bytes) 000489 1713294366.814608 mapread 0x013632 thru 0x017d6a (0x4739 bytes) 000490 1713294366.815166 mapread 0x0060ee thru 0x006314 (0x227 bytes) 000492 1713294366.815983 trunc from 0x017d6b to 0x033721 (0x1b9b7 bytes) 000496 1713294366.824223 MIRROR_RESYNC 10 mirrors 000499 1713294367.169048 mapwrite 0x03fe37 thru 0x03ffff (0x1c9 bytes) 000500 1713294367.206558 MIRROR_RESYNC 10 mirrors 000503 1713294367.488688 MIRROR_EXTEND to 11 mirrors 000504 1713294367.664551 write 0x03bd24 thru 0x03ffff (0x42dc bytes) 000506 1713294367.691832 mapread 0x0188da thru 0x0260fb (0xd822 bytes) 000508 1713294367.700472 mapwrite 0x02c84c thru 0x02f517 (0x2ccc bytes) 000510 1713294367.730959 MIRROR_EXTEND to 12 mirrors 000511 1713294367.888669 read 0x02af87 thru 0x036ac4 (0xbb3e bytes) 000512 1713294367.893630 write 0x021587 thru 0x0281f4 (0x6c6e bytes) 000513 1713294367.920074 write 0x036380 thru 0x03ffff (0x9c80 bytes) 000514 1713294367.925575 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000515 1713294367.989944 write 0x02b174 thru 0x03435b (0x91e8 bytes) 000518 1713294367.993783 trunc from 0x040000 to 0x00dfa3 (0x3205c bytes) 000519 1713294368.003119 mapread 0x009951 thru 0x00d756 (0x3e06 bytes) 000520 1713294368.006408 mapwrite 0x014188 thru 0x022411 (0xe28a bytes) 000521 1713294368.036975 MIRROR_RESYNC 11 mirrors 000522 1713294368.423246 read 0x004092 thru 0x008f2d (0x4e9c bytes) 000523 1713294368.425720 trunc from 0x022412 to 0x038974 (0x16563 bytes) 000526 1713294368.451709 mapwrite 0x0190be thru 0x021ad1 (0x8a14 bytes) 000527 1713294368.481816 mapwrite 0x034769 thru 0x03ad5a (0x65f2 bytes) 000528 1713294368.511485 write 0x026fb5 thru 0x03519a (0xe1e6 bytes) 000532 1713294368.514846 read 0x038144 thru 0x03ad5a (0x2c17 bytes) 000534 1713294368.517047 MIRROR_EXTEND to 12 mirrors 000535 1713294368.671449 read 0x01b791 thru 0x020052 (0x48c2 bytes) 000537 1713294368.678427 mapwrite 0x026450 thru 0x0278ad (0x145e bytes) 000538 1713294368.739428 mapread 0x009b8a thru 0x00f07e (0x54f5 bytes) 000540 1713294368.743558 read 0x002d86 thru 0x008712 (0x598d bytes) 000541 1713294368.747459 read 0x011543 thru 0x01c023 (0xaae1 bytes) 000542 1713294368.750213 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000544 1713294368.807540 trunc from 0x03ad5b to 0x0098bb (0x3149f bytes) 000545 1713294368.815542 read 0x004544 thru 0x0098bb (0x5378 bytes) 000551 1713294368.819086 write 0x033ff7 thru 0x03a46b (0x6475 bytes) 000552 1713294368.819878 read 0x01bf2b thru 0x0279e5 (0xbabb bytes) 000554 1713294368.822860 trunc from 0x03a46c to 0x011f01 (0x2856a bytes) 000556 1713294368.826972 trunc from 0x011f02 to 0x03bb7b (0x29c7a bytes) 000559 1713294368.832122 trunc from 0x03bb7c to 0x031a05 (0xa176 bytes) 000560 1713294368.836815 mapwrite 0x013ea2 thru 0x01d908 (0x9a67 bytes) 000561 1713294368.870570 trunc from 0x031a06 to 0x022ab8 (0xef4d bytes) 000563 1713294368.876169 mapread 0x0061cf thru 0x010497 (0xa2c9 bytes) 000566 1713294368.877234 trunc from 0x022ab9 to 0x01a94d (0x816b bytes) 000568 1713294368.882002 trunc from 0x01a94e to 0x003a30 (0x16f1d bytes) 000573 1713294368.887164 mapwrite 0x002cc6 thru 0x00b0cc (0x8407 bytes) 000575 1713294368.919292 mapwrite 0x00d123 thru 0x010bb9 (0x3a97 bytes) 000576 1713294368.945375 mapread 0x005660 thru 0x009558 (0x3ef9 bytes) 000577 1713294368.945943 read 0x0012b5 thru 0x00931a (0x8066 bytes) 000580 1713294368.946950 trunc from 0x010bba to 0x00fcdd (0xedc bytes) 000582 1713294368.951687 trunc from 0x00fcde to 0x008579 (0x7764 bytes) 000584 1713294368.957127 mapread 0x003d34 thru 0x008579 (0x4846 bytes) 000585 1713294368.959395 MIRROR_RESYNC 11 mirrors 000586 1713294369.342989 mapwrite 0x0244b5 thru 0x0247de (0x32a bytes) 000591 1713294369.396524 read 0x0161ac thru 0x01f74c (0x95a1 bytes) 000597 1713294369.402595 trunc from 0x0247df to 0x018c9e (0xbb40 bytes) 000598 1713294369.412858 MIRROR_EXTEND to 12 mirrors 000603 1713294369.565716 write 0x00a6f1 thru 0x01625d (0xbb6d bytes) 000605 1713294369.590576 mapwrite 0x005621 thru 0x00eee6 (0x98c6 bytes) 000608 1713294369.619383 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000612 1713294369.690678 MIRROR_EXTEND to 12 mirrors 000614 1713294369.848952 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000616 1713294369.918510 MIRROR_RESYNC 11 mirrors 000618 1713294370.354944 mapread 0x00e40c thru 0x018c9e (0xa893 bytes) 000620 1713294370.362871 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000622 1713294370.461364 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000623 1713294370.554772 MIRROR_EXTEND to 10 mirrors 000624 1713294370.726622 mapwrite 0x01e454 thru 0x01f0fe (0xcab bytes) 000625 1713294370.771232 mapread 0x009563 thru 0x01556a (0xc008 bytes) 000627 1713294370.777654 mapwrite 0x020d50 thru 0x029fe8 (0x9299 bytes) 000629 1713294370.813733 mapwrite 0x000aa1 thru 0x00b29b (0xa7fb bytes) 000632 1713294370.846232 trunc from 0x029fe9 to 0x03b24f (0x11267 bytes) 000634 1713294370.855588 MIRROR_EXTEND to 11 mirrors 000635 1713294371.014694 MIRROR_RESYNC 11 mirrors 000636 1713294371.406717 write 0x002479 thru 0x011159 (0xece1 bytes) 000637 1713294371.428874 mapread 0x017381 thru 0x0224d6 (0xb156 bytes) 000638 1713294371.436441 mapread 0x01f202 thru 0x028614 (0x9413 bytes) 000639 1713294371.437139 write 0x003945 thru 0x0107d0 (0xce8c bytes) 000641 1713294371.438546 trunc from 0x03b250 to 0x002f29 (0x38326 bytes) 000646 1713294371.447631 trunc from 0x002f2a to 0x0231f0 (0x202c7 bytes) 000649 1713294371.454092 read 0x01d1b3 thru 0x0206ae (0x34fc bytes) 000650 1713294371.456390 write 0x010ee1 thru 0x01c26c (0xb38c bytes) 000651 1713294371.461276 mapread 0x007787 thru 0x015c75 (0xe4ef bytes) 000655 1713294371.465673 mapwrite 0x024d5f thru 0x03000e (0xb2b0 bytes) 000658 1713294371.499078 mapwrite 0x03832a thru 0x03a790 (0x2467 bytes) 000659 1713294371.533533 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000660 1713294371.602262 write 0x01713e thru 0x023861 (0xc724 bytes) 000662 1713294371.609109 MIRROR_RESYNC 10 mirrors 000663 1713294372.048371 trunc from 0x03a791 to 0x01d583 (0x1d20d bytes) 000664 1713294372.079421 mapwrite 0x019d9d thru 0x01ba77 (0x1cdb bytes) 000666 1713294372.110102 mapwrite 0x027c86 thru 0x02cab0 (0x4e2b bytes) 000667 1713294372.144498 write 0x02f85b thru 0x03285e (0x3004 bytes) 000668 1713294372.145206 trunc from 0x03285f to 0x0211ac (0x116b2 bytes) 000669 1713294372.152215 read 0x01f627 thru 0x0211ac (0x1b86 bytes) 000671 1713294372.155152 mapwrite 0x016595 thru 0x0172f3 (0xd5f bytes) 000672 1713294372.177244 mapread 0x012ec2 thru 0x0198a3 (0x69e2 bytes) 000673 1713294372.182947 MIRROR_RESYNC 10 mirrors 000674 1713294372.602243 mapread 0x016f80 thru 0x01c0ef (0x5170 bytes) 000675 1713294372.608435 trunc from 0x0211ad to 0x039cad (0x18b01 bytes) 000676 1713294372.636742 mapread 0x00a7b6 thru 0x00fe44 (0x568f bytes) 000677 1713294372.641532 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000679 1713294372.720310 mapread 0x012c32 thru 0x020d8d (0xe15c bytes) 000680 1713294372.733387 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000682 1713294372.804156 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000683 1713294372.882330 mapread 0x00998a thru 0x00d761 (0x3dd8 bytes) 000684 1713294372.892836 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000686 1713294372.970422 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000688 1713294373.032822 trunc from 0x039cae to 0x016782 (0x2352b bytes) 000689 1713294373.037874 mapread 0x012dd7 thru 0x016782 (0x39ac bytes) 000690 1713294373.045403 mapwrite 0x01ef1d thru 0x02484c (0x5930 bytes) 000691 1713294373.091276 mapwrite 0x027926 thru 0x0296b9 (0x1d94 bytes) 000694 1713294373.127755 write 0x027935 thru 0x031a5f (0xa12b bytes) 000701 1713294373.132773 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000702 1713294373.212712 mapwrite 0x03b46e thru 0x03ffff (0x4b92 bytes) 000703 1713294373.253785 mapwrite 0x010f6a thru 0x01e23f (0xd2d6 bytes) 000706 1713294373.283837 read 0x005386 thru 0x00eb82 (0x97fd bytes) 000707 1713294373.286357 read 0x0341ae thru 0x03ffff (0xbe52 bytes) 000711 1713294373.293566 MIRROR_EXTEND to 5 mirrors 000712 1713294373.485287 write 0x0342ec thru 0x03b402 (0x7117 bytes) 000713 1713294373.517439 MIRROR_EXTEND to 6 mirrors 000714 1713294373.716337 trunc from 0x040000 to 0x0240fa (0x1bf05 bytes) 000715 1713294373.747554 read 0x017afb thru 0x01e7aa (0x6cb0 bytes) 000716 1713294373.751075 trunc from 0x0240fb to 0x03c410 (0x18316 bytes) 000720 1713294373.759965 read 0x006f81 thru 0x016968 (0xf9e8 bytes) 000721 1713294373.765032 trunc from 0x03c411 to 0x02a200 (0x12210 bytes) 000722 1713294373.773000 read 0x00000f thru 0x00f1c9 (0xf1bb bytes) 000723 1713294373.773507 write 0x000401 thru 0x00f35d (0xef5d bytes) 000724 1713294373.774398 read 0x01f946 thru 0x020000 (0x6bb bytes) 000725 1713294373.774696 mapwrite 0x02c02f thru 0x03985e (0xd830 bytes) 000726 1713294373.816277 mapwrite 0x000291 thru 0x00d151 (0xcec1 bytes) 000730 1713294373.842357 write 0x01bac6 thru 0x01da6b (0x1fa6 bytes) 000732 1713294373.843383 write 0x00f477 thru 0x01024a (0xdd4 bytes) 000738 1713294373.845680 write 0x02014f thru 0x020788 (0x63a bytes) 000739 1713294373.846309 mapwrite 0x03978d thru 0x03e532 (0x4da6 bytes) 000740 1713294373.882442 mapread 0x032bc9 thru 0x038065 (0x549d bytes) 000741 1713294373.883064 trunc from 0x03e533 to 0x03cb8f (0x19a3 bytes) 000743 1713294373.889472 trunc from 0x03cb90 to 0x026354 (0x1683b bytes) 000744 1713294373.895405 trunc from 0x026355 to 0x006a2b (0x1f929 bytes) 000747 1713294373.900555 read 0x00560d thru 0x006a2b (0x141f bytes) 000748 1713294373.900929 write 0x035798 thru 0x03c66a (0x6ed3 bytes) 000749 1713294373.901751 mapwrite 0x03f272 thru 0x03ffff (0xd8e bytes) 000750 1713294373.935701 MIRROR_RESYNC 6 mirrors 000754 1713294374.188690 write 0x02635d thru 0x0305c8 (0xa26c bytes) 000760 1713294374.217815 trunc from 0x040000 to 0x033ed1 (0xc12e bytes) 000762 1713294374.227016 MIRROR_EXTEND to 7 mirrors 000764 1713294374.413002 read 0x02c061 thru 0x033ed1 (0x7e71 bytes) 000766 1713294374.421070 mapwrite 0x03022e thru 0x037335 (0x7108 bytes) 000767 1713294374.482598 write 0x004642 thru 0x01180e (0xd1cd bytes) 000769 1713294374.491604 mapwrite 0x016026 thru 0x016f35 (0xf10 bytes) 000770 1713294374.517614 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000771 1713294374.586335 read 0x00283e thru 0x0068ba (0x407d bytes) 000773 1713294374.589707 trunc from 0x037336 to 0x009fbc (0x2d379 bytes) 000775 1713294374.596608 read 0x006203 thru 0x009fbc (0x3dba bytes) 000777 1713294374.601689 MIRROR_RESYNC 6 mirrors 000781 1713294374.920920 mapwrite 0x013d1b thru 0x020aef (0xcdd5 bytes) 000783 1713294375.009624 trunc from 0x020af0 to 0x037a42 (0x16f53 bytes) 000785 1713294375.021450 MIRROR_RESYNC 6 mirrors 000787 1713294375.359992 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000788 1713294375.463305 mapwrite 0x03f424 thru 0x03ffff (0xbdc bytes) 000794 1713294375.529840 MIRROR_RESYNC 5 mirrors 000795 1713294375.784950 mapwrite 0x00f82e thru 0x014fea (0x57bd bytes) 000797 1713294375.867177 read 0x0231ef thru 0x030078 (0xce8a bytes) 000798 1713294375.873872 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000799 1713294375.957036 trunc from 0x040000 to 0x0013f0 (0x3ec0f bytes) 000800 1713294375.973816 write 0x016c2f thru 0x01ecd2 (0x80a4 bytes) 000803 1713294375.975582 mapwrite 0x002ccb thru 0x0070fb (0x4431 bytes) 000804 1713294376.007636 trunc from 0x01ecd3 to 0x037216 (0x18544 bytes) 000805 1713294376.013056 mapread 0x01c2b9 thru 0x01d2f3 (0x103b bytes) 000806 1713294376.013660 read 0x02ee38 thru 0x037216 (0x83df bytes) 000807 1713294376.016746 mapread 0x01be04 thru 0x01e723 (0x2920 bytes) 000809 1713294376.017647 write 0x035b70 thru 0x03ebbb (0x904c bytes) 000812 1713294376.019090 read 0x032d3d thru 0x03ebbb (0xbe7f bytes) 000814 1713294376.022229 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000816 1713294376.102693 trunc from 0x03ebbc to 0x02307f (0x1bb3c bytes) 000821 1713294376.111165 mapwrite 0x0245ef thru 0x0283ae (0x3dc0 bytes) 000822 1713294376.153845 mapwrite 0x03be0e thru 0x03f7a7 (0x399a bytes) 000824 1713294376.195559 mapread 0x0263d7 thru 0x027e38 (0x1a62 bytes) 000826 1713294376.196537 read 0x039562 thru 0x03f7a7 (0x6246 bytes) 000827 1713294376.200064 MIRROR_RESYNC 3 mirrors 000830 1713294376.413626 mapread 0x00a6d0 thru 0x00c87e (0x21af bytes) 000831 1713294376.420518 read 0x03e14d thru 0x03f7a7 (0x165b bytes) 000835 1713294376.422669 mapread 0x032a67 thru 0x03f2ce (0xc868 bytes) 000836 1713294376.428936 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000837 1713294376.519754 write 0x00ce72 thru 0x01bfee (0xf17d bytes) 000838 1713294376.544060 write 0x010c2b thru 0x01324e (0x2624 bytes) 000841 1713294376.545277 write 0x01b3d4 thru 0x02a24a (0xee77 bytes) 000844 1713294376.548824 mapread 0x0085dd thru 0x00991e (0x1342 bytes) 000845 1713294376.552854 mapwrite 0x012608 thru 0x020f90 (0xe989 bytes) 000847 1713294376.582161 MIRROR_EXTEND to 3 mirrors 000848 1713294376.785003 mapwrite 0x02e7d2 thru 0x0390de (0xa90d bytes) 000849 1713294376.872317 MIRROR_EXTEND to 4 mirrors 000850 1713294377.052250 read 0x023929 thru 0x0257d3 (0x1eab bytes) 000851 1713294377.055737 trunc from 0x03f7a8 to 0x030f9e (0xe809 bytes) 000854 1713294377.096369 mapread 0x02c47a thru 0x030f9e (0x4b25 bytes) 000855 1713294377.102040 mapwrite 0x006828 thru 0x00f73f (0x8f18 bytes) 000856 1713294377.131678 read 0x013836 thru 0x01aaf7 (0x72c2 bytes) 000858 1713294377.135616 mapread 0x00ce3f thru 0x01bc13 (0xedd5 bytes) 000861 1713294377.136862 write 0x014511 thru 0x015a62 (0x1552 bytes) 000863 1713294377.137806 write 0x024971 thru 0x0304b7 (0xbb47 bytes) 000865 1713294377.139108 mapread 0x00eb04 thru 0x0121ff (0x36fc bytes) 000866 1713294377.140843 mapwrite 0x03195d thru 0x039b5d (0x8201 bytes) 000867 1713294377.185306 MIRROR_EXTEND to 5 mirrors 000868 1713294377.364579 trunc from 0x039b5e to 0x01e2a9 (0x1b8b4 bytes) 000869 1713294377.394263 read 0x0198b2 thru 0x01e2a9 (0x49f8 bytes) 000870 1713294377.399296 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000871 1713294377.464194 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000872 1713294377.529351 mapwrite 0x02bc54 thru 0x03547b (0x9828 bytes) 000875 1713294377.567618 mapwrite 0x026f5a thru 0x031a88 (0xab2f bytes) 000876 1713294377.606963 MIRROR_EXTEND to 4 mirrors 000881 1713294377.794544 mapread 0x027f57 thru 0x03513d (0xd1e7 bytes) 000883 1713294377.808627 mapread 0x01b63b thru 0x01bcaa (0x670 bytes) 000886 1713294377.818328 trunc from 0x03547c to 0x029fa4 (0xb4d7 bytes) 000887 1713294377.849939 write 0x00e6a0 thru 0x01ae14 (0xc775 bytes) 000888 1713294377.855151 write 0x00206c thru 0x00e72a (0xc6bf bytes) 000889 1713294377.859413 mapwrite 0x003315 thru 0x013047 (0xfd33 bytes) 000890 1713294377.896646 mapwrite 0x0238a7 thru 0x02d29e (0x99f8 bytes) 000891 1713294377.958107 read 0x006669 thru 0x00a0af (0x3a47 bytes) 000892 1713294377.958540 trunc from 0x02d29f to 0x01ee8a (0xe414 bytes) 000893 1713294377.970746 mapwrite 0x00faec thru 0x013825 (0x3d3a bytes) 000895 1713294378.001821 trunc from 0x01ee8b to 0x02e66f (0xf7e5 bytes) 000896 1713294378.006789 read 0x0001ad thru 0x002fab (0x2dff bytes) 000898 1713294378.010411 trunc from 0x02e670 to 0x009d51 (0x2491e bytes) 000899 1713294378.016412 mapread 0x0066c7 thru 0x009d51 (0x368b bytes) 000900 1713294378.017206 mapwrite 0x00d9c8 thru 0x00e301 (0x93a bytes) 000901 1713294378.048327 read 0x0009e8 thru 0x00e301 (0xd91a bytes) 000904 1713294378.050993 mapread 0x0069ea thru 0x00e301 (0x7918 bytes) 000905 1713294378.051564 read 0x00d9f1 thru 0x00e301 (0x911 bytes) 000908 1713294378.052587 mapread 0x00b5a4 thru 0x00e301 (0x2d5e bytes) 000910 1713294378.053459 trunc from 0x00e302 to 0x02fc10 (0x2190f bytes) 000913 1713294378.061655 mapread 0x00aa0d thru 0x00ea18 (0x400c bytes) 000914 1713294378.064221 MIRROR_EXTEND to 5 mirrors 000915 1713294378.241555 read 0x018113 thru 0x0240fa (0xbfe8 bytes) 000916 1713294378.247593 mapwrite 0x00a9f6 thru 0x01560e (0xac19 bytes) 000919 1713294378.336554 write 0x0384f0 thru 0x03ffff (0x7b10 bytes) 000920 1713294378.340065 MIRROR_EXTEND to 6 mirrors 000921 1713294378.525333 trunc from 0x040000 to 0x000260 (0x3fd9f bytes) 000922 1713294378.554485 read 0x0001d2 thru 0x000260 (0x8f bytes) 000924 1713294378.559139 trunc from 0x000261 to 0x025252 (0x24ff2 bytes) 000926 1713294378.569047 trunc from 0x025253 to 0x025a9a (0x848 bytes) 000927 1713294378.575210 mapwrite 0x00f740 thru 0x015cc4 (0x6585 bytes) 000928 1713294378.606043 mapwrite 0x0027e5 thru 0x003f22 (0x173e bytes) 000929 1713294378.634999 read 0x00a0f3 thru 0x0183f8 (0xe306 bytes) 000931 1713294378.639325 MIRROR_EXTEND to 7 mirrors 000932 1713294378.812608 write 0x00bf58 thru 0x019b2f (0xdbd8 bytes) 000934 1713294378.843093 trunc from 0x025a9b to 0x031825 (0xbd8b bytes) 000935 1713294378.855452 MIRROR_RESYNC 7 mirrors 000936 1713294379.257281 write 0x021398 thru 0x027bfe (0x6867 bytes) 000937 1713294379.288468 mapread 0x02143f thru 0x030060 (0xec22 bytes) 000938 1713294379.294390 mapread 0x02e912 thru 0x02efc8 (0x6b7 bytes) 000939 1713294379.295313 write 0x001ef7 thru 0x00bc6b (0x9d75 bytes) 000940 1713294379.299251 mapread 0x012517 thru 0x01bd99 (0x9883 bytes) 000941 1713294379.305327 write 0x0143da thru 0x01b6fe (0x7325 bytes) 000942 1713294379.306125 mapread 0x01e4a6 thru 0x027012 (0x8b6d bytes) 000944 1713294379.307191 read 0x002c75 thru 0x00e539 (0xb8c5 bytes) 000947 1713294379.308296 write 0x01eb2b thru 0x021918 (0x2dee bytes) 000951 1713294379.310514 mapwrite 0x00c95c thru 0x0191d5 (0xc87a bytes) 000952 1713294379.337562 mapread 0x029199 thru 0x029b05 (0x96d bytes) 000954 1713294379.338433 write 0x026fb6 thru 0x02744b (0x496 bytes) 000955 1713294379.338971 trunc from 0x031826 to 0x023448 (0xe3dd bytes) 000956 1713294379.347039 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000957 1713294379.425098 mapread 0x00b16a thru 0x012710 (0x75a7 bytes) 000958 1713294379.432842 mapread 0x0059a3 thru 0x00e117 (0x8775 bytes) 000959 1713294379.437104 mapread 0x00b69f thru 0x01b22b (0xfb8d bytes) 000960 1713294379.443343 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000962 1713294379.513475 read 0x001b55 thru 0x00c473 (0xa91f bytes) 000964 1713294379.519192 read 0x022618 thru 0x023448 (0xe31 bytes) 000969 1713294379.528204 mapwrite 0x03ec27 thru 0x03ffff (0x13d9 bytes) 000971 1713294379.577603 mapread 0x015836 thru 0x021a59 (0xc224 bytes) 000972 1713294379.578312 write 0x0055a5 thru 0x01492c (0xf388 bytes) 000974 1713294379.579696 read 0x00b430 thru 0x00d928 (0x24f9 bytes) 000976 1713294379.580504 write 0x03bf08 thru 0x03d237 (0x1330 bytes) 000977 1713294379.581139 read 0x026394 thru 0x03132f (0xaf9c bytes) 000978 1713294379.581564 trunc from 0x040000 to 0x00b99b (0x34664 bytes) 000979 1713294379.587701 mapwrite 0x01e248 thru 0x02c4d6 (0xe28f bytes) 000981 1713294379.630664 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000983 1713294379.707568 mapread 0x0067c6 thru 0x00bbb2 (0x53ed bytes) 000984 1713294379.715016 mapwrite 0x021d87 thru 0x027514 (0x578e bytes) 000985 1713294379.743083 write 0x0071f0 thru 0x012a59 (0xb86a bytes) 000986 1713294379.747454 MIRROR_EXTEND to 5 mirrors 000988 1713294379.951747 read 0x0168d7 thru 0x01c5f5 (0x5d1f bytes) 000989 1713294379.958402 mapread 0x00e479 thru 0x013b56 (0x56de bytes) 000990 1713294379.970516 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000992 1713294380.031973 read 0x009341 thru 0x010e5b (0x7b1b bytes) 000993 1713294380.040642 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000994 1713294380.114091 read 0x005828 thru 0x014f62 (0xf73b bytes) 000997 1713294380.119323 write 0x01ccc0 thru 0x029ed2 (0xd213 bytes) 001000 1713294380.130535 MIRROR_RESYNC 3 mirrors All operations completed A-OK! PASS 70a (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 15:06:23 (1713294383) Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre2 Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2150144 .. Extending file size to 6620416 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7137152 .. failed Extending file size to 4837888 .. Extending file size to 6304224 .. Extending file size to 7882272 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7811040 .. Extending file size to 8077440 .. Extending file size to 8198080 .. Extending file size to 8394336 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9116096 .. Extending file size to 8411424 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9163456 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 2877 2878 2879 2880 2882 failed 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) 192.168.204.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg429-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:07:30 (1713294450) Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre2 Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6461312 .. Extending file size to 3960064 .. Extending file size to 7229088 .. Extending file size to 7280704 .. Extending file size to 7964864 .. Extending file size to 7543232 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8299264 .. Extending file size to 7828768 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8509568 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9030272 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8431616 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8872064 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9362688 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 5584 5585 5586 5587 5590 failed 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) 192.168.204.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg429-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:08:35 (1713294515) Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4736 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4736 2203776 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0335357 s, 62.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0742292 s, 28.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:08:41 (1713294521) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0292537 s, 71.7 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 15:08:44 (1713294524) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x823:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x823:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x824:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x81e:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:08:47 (1713294527) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0290061 s, 72.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x825:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x81f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x820:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:08:51 (1713294531) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0265488 s, 79.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x826:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x827:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x821:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x826:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x827:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x821:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x822:0x0] } PASS 204b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:08:54 (1713294534) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0319498 s, 65.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x828:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x823:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x828:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x823:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x824:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:09:19 (1713294559) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193137 s, 54.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x829:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x826:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x825:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x829:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x826:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x825:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x82a:0x0] } PASS 204d (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:10:02 (1713294602) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x827:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x828:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0260733 s, 80.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x827:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x828:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x82b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x827:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x828:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x82b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:10:47 (1713294647) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg429-server mds-ost sync done. Creating new pool oleg429-server: Pool lustre.test_204f created Adding targets to pool oleg429-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg429-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0259569 s, 80.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x829:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x82c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x829:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x82c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg429-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg429-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg429-server: Pool lustre.test_204f destroyed PASS 204f (50s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 15:11:38 (1713294698) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:11:42 (1713294702) oleg429-server: Pool lustre.test_206 created Adding targets to pool oleg429-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg429-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg429-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg429-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg429-server: Pool lustre.test_206 destroyed PASS 206 (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:11:54 (1713294714) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198562 s, 52.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00900389 s, 116 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x5445a311 Mirror 3: 0x5445a311 PASS 207 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:11:58 (1713294718) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:12:00 (1713294720) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:12:03 (1713294723) Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0219173 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x23e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2520000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2520000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2520000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12d0000 Read this (4 bytes): BBBB 192.168.204.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg429-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:12:18 (1713294738) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0030025 s, 2.7 MB/s the very first read read with warm cache PASS 209b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 15:12:21 (1713294741) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107377 s, 97.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x4c1 lmm_fid: [0x200000404:0x4c1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x4c1 lmm_fid: [0x200000404:0x4c1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x833:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835137 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-16 15:12:21.000000000 -0400 Modify: 2024-04-16 15:12:21.000000000 -0400 Change: 2024-04-16 15:12:22.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:12:24 (1713294744) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 847 sec ============ 15:12:25 (1713294745) === sanity-flr: start cleanup 15:12:26 (1713294746) === === sanity-flr: finish cleanup 15:12:26 (1713294746) ===