-----============= acceptance-small: sanity-flr ============----- Thu Apr 18 20:16:12 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:16:15 (1713485775) === oleg441-client.virtnet: executing check_config_client /mnt/lustre oleg441-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg441-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a4b8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a4b8800.idle_timeout=debug disable quota as required oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:16:21 (1713485781) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6918] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 20:16:22 (1713485782) PASS 0a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:16:27 (1713485787) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:16:30 (1713485790) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 20:16:32 (1713485792) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:16:38 (1713485798) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:16:40 (1713485800) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 20:16:42 (1713485802) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a4b8800.stats=clear osc.lustre-OST0001-osc-ffff88012a4b8800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:16:46 (1713485806) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:16:49 (1713485809) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:16:52 (1713485812) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 20:16:55 (1713485815) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 20:16:58 (1713485818) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:17:01 (1713485821) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:17:03 (1713485823) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:17:07 (1713485827) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 20:17:10 (1713485830) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:17:13 (1713485833) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.544633 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0502076 s, 20.9 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:17:18 (1713485838) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00714978 s, 147 MB/s PASS 22 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:17:21 (1713485841) Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4b8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4b8800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4b8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4b8800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:17:44 (1713485864) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0747924 s, 70.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4b8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4b8800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:17:55 (1713485875) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4b8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4b8800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4b8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4b8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:18:11 (1713485891) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.743148 s, 70.5 MB/s -rw-r--r-- 1 root root 52428800 Apr 18 20:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.167s user 0m0.000s sys 0m0.136s Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4b8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4b8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.184s user 0m0.002s sys 0m0.150s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4b8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4b8800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.182s user 0m0.001s sys 0m0.140s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a4b8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4b8800.ost_server_uuid in FULL state after 0 sec PASS 33b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:18:32 (1713485912) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:18:34 (1713485914) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:18:36 (1713485916) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 20:18:39 (1713485919) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 20:18:42 (1713485922) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.114923 s, 109 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:18:46 (1713485926) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00851694 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0066372 s, 158 MB/s PASS 36b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:18:49 (1713485929) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:18:53 (1713485933) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00061404 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000475843 s, 2.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000614183 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00056972 s, 1.8 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 20:18:57 (1713485937) create test files with size 15755671 .. 4864a7430b9b5fa995f4fb0da37b9f47 - df3c3564ecdb4d52c86c4f73da23f4ca - c51d65a5d9c30391de99eb47180cf359 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 20:19:03 (1713485943) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:19:07 (1713485947) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:19:10 (1713485950) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191343 s, 54.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x42:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00780394 s, 134 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 20:19:13 (1713485953) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0231772 s, 90.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0753884 s, 55.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.042311 s, 49.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 20:19:17 (1713485957) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 20:19:19 (1713485959) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:19:21 (1713485961) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00819752 s, 128 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 20:19:24 (1713485964) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:19:27 (1713485967) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:19:30 (1713485970) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0519458 s, 202 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:19:34 (1713485974) llite.lustre-ffff8800aaf6e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0666237 s, 157 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800b6277800.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:19:38 (1713485978) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 20:19:45 (1713485985) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 20:19:52 (1713485992) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 20:19:54 (1713485994) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0621664 s, 50.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0447295 s, 70.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 3552731b0e2db1471d3c1179f7f190e7 - mirror 1 checksum: 23e4beca3c7d6960efe4a6a1e4cad2c3 - mirror 2 checksum: 3552731b0e2db1471d3c1179f7f190e7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 20:19:55 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 20:19:55 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3552731b0e2db1471d3c1179f7f190e7 - mirror 1 checksum: 23e4beca3c7d6960efe4a6a1e4cad2c3 - mirror 2 checksum: 23e4beca3c7d6960efe4a6a1e4cad2c3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 20:19:55 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 20:19:55 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:19:58 (1713485998) Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server 20:20:00 (1713486000) shut down Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 20:20:12 (1713486012) targets are mounted 20:20:12 (1713486012) facet_failover done oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00780271 s, 134 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } PASS 50A (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:20:18 (1713486018) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 20:20:20 (1713486020) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:20:23 (1713486023) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00195636 s, 8.4 MB/s PASS 50c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 20:20:26 (1713486026) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:20:28 (1713486028) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:20:30 (1713486030) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:20:35 (1713486035) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:20:54 (1713486054) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:21:15 (1713486075) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:21:35 (1713486095) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 20:21:40 (1713486100) mirror create pid 26630 mirror split pid 26631 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26630 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26631 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 20:22:44 (1713486164) Seed set to 6164 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x25c76 000001 1713486164.917494 trunc from 00000000 to 0x025c75 (0x25c76 bytes) 000003 1713486164.924139 mapwrite 0x02ebcc thru 0x031c89 (0x30be bytes) 000004 1713486164.975430 MIRROR_EXTEND to 2 mirrors 000005 1713486165.124327 mapwrite 0x02aed7 thru 0x032b1d (0x7c47 bytes) 000012 1713486165.179373 trunc from 0x032b1e to 0x016560 (0x1c5bd bytes) 000014 1713486165.185178 read 0x0129cf thru 0x016560 (0x3b92 bytes) 000015 1713486165.187986 read 0x006d89 thru 0x009140 (0x23b8 bytes) 000016 1713486165.193209 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000019 1713486165.229670 mapread 0x00587e thru 0x008ed2 (0x3655 bytes) 000021 1713486165.236276 mapread 0x0151e6 thru 0x016560 (0x137b bytes) 000025 1713486165.242685 MIRROR_EXTEND to 2 mirrors 000026 1713486165.368243 write 0x01d453 thru 0x021855 (0x4403 bytes) 000027 1713486165.385662 mapread 0x007265 thru 0x00c76b (0x5507 bytes) 000028 1713486165.391364 read 0x017443 thru 0x01fc3e (0x87fc bytes) 000029 1713486165.394007 mapwrite 0x005d82 thru 0x01531f (0xf59e bytes) 000030 1713486165.416275 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000031 1713486165.454009 MIRROR_EXTEND to 2 mirrors 000033 1713486165.574419 mapwrite 0x0224c3 thru 0x022e49 (0x987 bytes) 000034 1713486165.629419 read 0x011ee8 thru 0x017dd1 (0x5eea bytes) 000035 1713486165.633594 write 0x00d19f thru 0x01d06d (0xfecf bytes) 000036 1713486165.641783 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) truncating to largest ever: 0x3630a 000037 1713486165.682538 trunc from 0x022e4a to 0x036309 (0x134c0 bytes) 000038 1713486165.686985 read 0x0083ac thru 0x0144d9 (0xc12e bytes) 000040 1713486165.690416 write 0x00af5c thru 0x00f91b (0x49c0 bytes) 000047 1713486165.693023 mapread 0x011720 thru 0x013b8b (0x246c bytes) 000049 1713486165.693796 read 0x0051c9 thru 0x0099de (0x4816 bytes) 000050 1713486165.699380 MIRROR_EXTEND to 2 mirrors 000051 1713486165.824323 mapwrite 0x01a9e8 thru 0x028650 (0xdc69 bytes) 000052 1713486165.876788 MIRROR_RESYNC 2 mirrors 000053 1713486165.991966 read 0x0236c4 thru 0x029a8c (0x63c9 bytes) 000057 1713486165.994818 mapwrite 0x02a3c7 thru 0x0384b6 (0xe0f0 bytes) 000060 1713486166.041611 mapread 0x01cbf9 thru 0x02123c (0x4644 bytes) 000061 1713486166.044990 write 0x016d5b thru 0x01a64e (0x38f4 bytes) 000063 1713486166.049480 MIRROR_EXTEND to 3 mirrors 000064 1713486166.170094 trunc from 0x0384b7 to 0x00f946 (0x28b70 bytes) 000065 1713486166.196189 mapread 0x00e121 thru 0x00f946 (0x1826 bytes) 000066 1713486166.199674 read 0x00885e thru 0x00f946 (0x70e9 bytes) 000068 1713486166.202423 trunc from 0x00f947 to 0x01de5e (0xe518 bytes) 000070 1713486166.207968 trunc from 0x01de5f to 0x00f66b (0xe7f3 bytes) 000071 1713486166.212503 mapwrite 0x019f1e thru 0x01b3ef (0x14d2 bytes) 000074 1713486166.232871 read 0x00ef06 thru 0x0149de (0x5ad9 bytes) 000075 1713486166.237704 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000076 1713486166.286395 trunc from 0x01b3f0 to 0x011d50 (0x969f bytes) 000077 1713486166.291143 read 0x010a68 thru 0x011d50 (0x12e9 bytes) 000079 1713486166.293571 mapread 0x004add thru 0x00e55c (0x9a80 bytes) 000081 1713486166.301159 mapwrite 0x01a47f thru 0x021c54 (0x77d6 bytes) 000083 1713486166.327707 MIRROR_EXTEND to 3 mirrors 000084 1713486166.441409 MIRROR_RESYNC 3 mirrors 000086 1713486166.540402 trunc from 0x021c55 to 0x025cfe (0x40aa bytes) 000087 1713486166.559530 mapwrite 0x0121c0 thru 0x019d39 (0x7b7a bytes) 000088 1713486166.581795 mapwrite 0x02d5e6 thru 0x03a1dd (0xcbf8 bytes) 000090 1713486166.611418 trunc from 0x03a1de to 0x00d6d9 (0x2cb04 bytes) 000095 1713486166.616958 read 0x001c04 thru 0x00d6d9 (0xbad6 bytes) 000096 1713486166.619981 trunc from 0x00d6da to 0x009c9f (0x3a3a bytes) 000097 1713486166.624573 mapwrite 0x03ab0c thru 0x03ffff (0x54f4 bytes) 000098 1713486166.650743 mapread 0x018c88 thru 0x026a74 (0xdded bytes) truncating to largest ever: 0x36557 000099 1713486166.657100 trunc from 0x040000 to 0x036556 (0x9aa9 bytes) 000100 1713486166.662227 read 0x01a479 thru 0x0210f7 (0x6c7f bytes) 000105 1713486166.665625 MIRROR_RESYNC 3 mirrors 000107 1713486166.802697 write 0x00192d thru 0x00ee24 (0xd4f8 bytes) 000108 1713486166.818925 mapread 0x002936 thru 0x003851 (0xf1c bytes) 000109 1713486166.819617 mapread 0x0295d7 thru 0x02ff15 (0x693f bytes) 000110 1713486166.823639 MIRROR_RESYNC 3 mirrors 000113 1713486166.939603 mapwrite 0x00296f thru 0x010880 (0xdf12 bytes) 000114 1713486166.996342 read 0x00fde2 thru 0x0168b7 (0x6ad6 bytes) 000117 1713486166.999913 mapwrite 0x02359c thru 0x02c998 (0x93fd bytes) 000119 1713486167.023827 mapread 0x01e98c thru 0x0234da (0x4b4f bytes) 000122 1713486167.025046 mapwrite 0x01cc5f thru 0x028c80 (0xc022 bytes) 000123 1713486167.045765 mapwrite 0x001b96 thru 0x01068b (0xeaf6 bytes) 000124 1713486167.065005 read 0x0303bd thru 0x036556 (0x619a bytes) 000126 1713486167.069814 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000127 1713486167.120708 mapread 0x0123d8 thru 0x01b99a (0x95c3 bytes) 000129 1713486167.126696 read 0x028943 thru 0x030cb8 (0x8376 bytes) 000132 1713486167.129871 read 0x01c793 thru 0x02c765 (0xffd3 bytes) 000134 1713486167.134920 MIRROR_RESYNC 2 mirrors 000136 1713486167.241493 mapread 0x01d451 thru 0x0232a8 (0x5e58 bytes) 000137 1713486167.247263 trunc from 0x036557 to 0x02fa11 (0x6b45 bytes) 000138 1713486167.268467 mapwrite 0x031ee8 thru 0x03fe1c (0xdf35 bytes) 000141 1713486167.294319 mapwrite 0x02bd44 thru 0x033c5b (0x7f18 bytes) 000143 1713486167.314100 trunc from 0x03fe1d to 0x02d872 (0x125aa bytes) 000144 1713486167.319307 read 0x0071b9 thru 0x00ac69 (0x3ab1 bytes) 000145 1713486167.323375 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000147 1713486167.359135 write 0x02ab1b thru 0x032649 (0x7b2f bytes) 000148 1713486167.361515 mapwrite 0x038b44 thru 0x03db52 (0x500f bytes) 000150 1713486167.387999 mapread 0x038477 thru 0x03882d (0x3b7 bytes) 000151 1713486167.388536 mapwrite 0x02c073 thru 0x033756 (0x76e4 bytes) 000152 1713486167.404250 trunc from 0x03db53 to 0x010f7e (0x2cbd4 bytes) 000153 1713486167.408555 read 0x00d3be thru 0x010f7e (0x3bc1 bytes) 000154 1713486167.410151 read 0x00a518 thru 0x010f7e (0x6a67 bytes) 000157 1713486167.413095 mapread 0x00e189 thru 0x010f7e (0x2df6 bytes) 000159 1713486167.413953 mapread 0x004dbc thru 0x005140 (0x385 bytes) 000164 1713486167.415577 mapwrite 0x035dd6 thru 0x0381ff (0x242a bytes) 000165 1713486167.440019 mapwrite 0x007b0e thru 0x008e7a (0x136d bytes) 000166 1713486167.453713 mapread 0x00bde7 thru 0x018e10 (0xd02a bytes) 000169 1713486167.454704 write 0x0195b5 thru 0x028902 (0xf34e bytes) 000171 1713486167.455666 mapread 0x00f059 thru 0x0177a0 (0x8748 bytes) 000173 1713486167.456486 read 0x02711a thru 0x02fe74 (0x8d5b bytes) 000175 1713486167.457202 mapread 0x037c4a thru 0x0381ff (0x5b6 bytes) 000176 1713486167.457680 mapwrite 0x023709 thru 0x02c91a (0x9212 bytes) 000177 1713486167.473777 read 0x006dca thru 0x0101b4 (0x93eb bytes) 000178 1713486167.474197 read 0x020bf0 thru 0x028fa8 (0x83b9 bytes) truncating to largest ever: 0x3831e 000179 1713486167.474493 trunc from 0x038200 to 0x03831d (0x11e bytes) 000183 1713486167.480057 write 0x03d2b8 thru 0x03ffff (0x2d48 bytes) 000184 1713486167.480650 write 0x022937 thru 0x02f470 (0xcb3a bytes) 000187 1713486167.481948 write 0x008794 thru 0x009cf3 (0x1560 bytes) 000190 1713486167.484745 MIRROR_EXTEND to 2 mirrors 000191 1713486167.608426 read 0x01510b thru 0x01de80 (0x8d76 bytes) 000202 1713486167.612880 mapwrite 0x035621 thru 0x03ffff (0xa9df bytes) 000204 1713486167.666117 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) truncating to largest ever: 0x390d8 000205 1713486167.699140 trunc from 0x040000 to 0x0390d7 (0x6f28 bytes) 000206 1713486167.707209 mapwrite 0x03dfa8 thru 0x03f533 (0x158c bytes) 000207 1713486167.733586 trunc from 0x03f534 to 0x02fb80 (0xf9b3 bytes) 000208 1713486167.737947 trunc from 0x02fb81 to 0x02165c (0xe524 bytes) 000209 1713486167.742403 write 0x01cc26 thru 0x02ad27 (0xe102 bytes) 000213 1713486167.749430 trunc from 0x02ad28 to 0x00c131 (0x1ebf6 bytes) 000215 1713486167.754217 mapread 0x0052e1 thru 0x00c131 (0x6e51 bytes) 000217 1713486167.757532 read 0x00a4e5 thru 0x00c131 (0x1c4d bytes) 000218 1713486167.757806 read 0x00a697 thru 0x00c131 (0x1a9b bytes) 000219 1713486167.758134 write 0x002708 thru 0x003115 (0xa0e bytes) 000220 1713486167.760409 mapread 0x0029f2 thru 0x005fca (0x35d9 bytes) 000221 1713486167.760796 mapwrite 0x03eb2a thru 0x03ec68 (0x13f bytes) 000223 1713486167.787470 trunc from 0x03ec69 to 0x017071 (0x27bf7 bytes) 000224 1713486167.792445 trunc from 0x017072 to 0x01af49 (0x3ed8 bytes) 000227 1713486167.797179 trunc from 0x01af4a to 0x00e97b (0xc5ce bytes) 000229 1713486167.802969 write 0x000fd8 thru 0x008235 (0x725e bytes) 000231 1713486167.804162 mapwrite 0x0368c9 thru 0x03ae4c (0x4584 bytes) 000234 1713486167.837137 trunc from 0x03ae4d to 0x0131c9 (0x27c83 bytes) 000235 1713486167.842103 mapread 0x0126b9 thru 0x0131c9 (0xb11 bytes) 000236 1713486167.845999 mapwrite 0x01befb thru 0x0202bf (0x43c5 bytes) 000237 1713486167.876249 trunc from 0x0202c0 to 0x037f98 (0x17cd9 bytes) 000238 1713486167.881163 trunc from 0x037f99 to 0x01167e (0x2691a bytes) 000244 1713486167.887637 trunc from 0x01167f to 0x03691a (0x2529c bytes) 000246 1713486167.893497 mapread 0x029cb8 thru 0x02f534 (0x587d bytes) 000249 1713486167.904209 read 0x011215 thru 0x0124cd (0x12b9 bytes) 000250 1713486167.908586 read 0x01cf30 thru 0x01ef18 (0x1fe9 bytes) 000251 1713486167.909017 mapread 0x018b7a thru 0x01c216 (0x369d bytes) 000255 1713486167.912469 MIRROR_EXTEND to 2 mirrors 000256 1713486168.037090 read 0x01387a thru 0x01808e (0x4815 bytes) 000258 1713486168.039662 mapwrite 0x01e3d3 thru 0x02a267 (0xbe95 bytes) 000259 1713486168.103861 MIRROR_RESYNC 2 mirrors 000260 1713486168.222488 mapread 0x01c288 thru 0x027b75 (0xb8ee bytes) 000261 1713486168.233942 read 0x03401e thru 0x03691a (0x28fd bytes) 000263 1713486168.236789 write 0x03e128 thru 0x03ffff (0x1ed8 bytes) 000264 1713486168.255409 mapread 0x000abc thru 0x002d27 (0x226c bytes) 000265 1713486168.259306 write 0x00df31 thru 0x012a8d (0x4b5d bytes) 000267 1713486168.261288 trunc from 0x040000 to 0x00582d (0x3a7d2 bytes) 000270 1713486168.267418 trunc from 0x00582e to 0x0210b7 (0x1b88a bytes) 000272 1713486168.271936 mapread 0x01887c thru 0x0210b7 (0x883c bytes) 000273 1713486168.277266 mapread 0x01e167 thru 0x0210b7 (0x2f51 bytes) 000275 1713486168.278044 write 0x029efa thru 0x02bee7 (0x1fee bytes) 000279 1713486168.281413 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000283 1713486168.318210 mapwrite 0x01e3f1 thru 0x025689 (0x7299 bytes) 000284 1713486168.349044 trunc from 0x02bee8 to 0x010c52 (0x1b295 bytes) 000285 1713486168.353386 read 0x004ef3 thru 0x008ae8 (0x3bf6 bytes) 000287 1713486168.356557 trunc from 0x010c53 to 0x01a554 (0x9902 bytes) 000290 1713486168.361610 read 0x001bbe thru 0x004bb0 (0x2ff3 bytes) 000291 1713486168.364804 mapwrite 0x013862 thru 0x01f208 (0xb9a7 bytes) 000293 1713486168.387599 mapread 0x00e2ca thru 0x01b233 (0xcf6a bytes) 000295 1713486168.388492 trunc from 0x01f209 to 0x033a58 (0x14850 bytes) 000297 1713486168.393039 trunc from 0x033a59 to 0x022c3c (0x10e1c bytes) 000298 1713486168.397913 write 0x01af2c thru 0x0224b1 (0x7586 bytes) 000299 1713486168.399922 trunc from 0x022c3d to 0x02ad14 (0x80d8 bytes) 000300 1713486168.403797 write 0x030327 thru 0x03d1f0 (0xceca bytes) 000302 1713486168.404711 read 0x0239f8 thru 0x02ba33 (0x803c bytes) 000309 1713486168.409213 mapread 0x03be9b thru 0x03d1f0 (0x1356 bytes) 000310 1713486168.409718 trunc from 0x03d1f1 to 0x003049 (0x3a1a7 bytes) 000311 1713486168.414806 read 0x0022dc thru 0x003049 (0xd6e bytes) 000312 1713486168.415153 mapwrite 0x03e9d9 thru 0x03ffff (0x1627 bytes) 000313 1713486168.442438 mapread 0x0305d3 thru 0x03ef20 (0xe94e bytes) 000314 1713486168.443040 mapread 0x03408b thru 0x03ab10 (0x6a86 bytes) 000315 1713486168.443542 read 0x01c11e thru 0x028d4c (0xcc2f bytes) 000316 1713486168.443889 trunc from 0x040000 to 0x00b265 (0x34d9a bytes) 000317 1713486168.448569 mapwrite 0x00b573 thru 0x018511 (0xcf9f bytes) 000318 1713486168.475154 trunc from 0x018512 to 0x00d8ac (0xac65 bytes) 000319 1713486168.479294 read 0x00860d thru 0x00d8ac (0x52a0 bytes) 000322 1713486168.480242 mapread 0x00021a thru 0x004324 (0x410b bytes) 000324 1713486168.481050 write 0x00e6f0 thru 0x01236e (0x3c7f bytes) 000325 1713486168.481618 trunc from 0x01236f to 0x00a38c (0x7fe2 bytes) 000326 1713486168.486188 trunc from 0x00a38d to 0x004713 (0x5c79 bytes) 000327 1713486168.490110 mapwrite 0x03f535 thru 0x03ffff (0xacb bytes) 000330 1713486168.512095 read 0x011ec2 thru 0x019f55 (0x8094 bytes) 000331 1713486168.512535 trunc from 0x040000 to 0x008555 (0x37aaa bytes) 000332 1713486168.517276 write 0x015dc9 thru 0x0194e9 (0x3721 bytes) 000333 1713486168.517977 read 0x005d3d thru 0x012f6c (0xd230 bytes) 000335 1713486168.520707 mapread 0x001300 thru 0x00b99c (0xa69d bytes) 000337 1713486168.521442 mapwrite 0x034972 thru 0x0379da (0x3069 bytes) 000339 1713486168.547308 MIRROR_EXTEND to 2 mirrors 000340 1713486168.670749 read 0x009e7d thru 0x012fe7 (0x916b bytes) 000343 1713486168.673946 write 0x03a722 thru 0x03cb4a (0x2429 bytes) 000344 1713486168.689405 read 0x013a90 thru 0x01d02a (0x959b bytes) 000345 1713486168.694081 mapread 0x03550c thru 0x03cb4a (0x763f bytes) 000346 1713486168.694597 read 0x02a0b4 thru 0x034f21 (0xae6e bytes) 000349 1713486168.695364 write 0x012807 thru 0x013331 (0xb2b bytes) 000353 1713486168.698034 MIRROR_RESYNC 2 mirrors 000357 1713486168.793299 mapread 0x00ce0c thru 0x01003f (0x3234 bytes) 000359 1713486168.798655 mapread 0x038651 thru 0x03cb4a (0x44fa bytes) 000361 1713486168.802572 MIRROR_EXTEND to 3 mirrors 000363 1713486168.922759 write 0x01ffd2 thru 0x0296b1 (0x96e0 bytes) 000364 1713486168.942477 mapwrite 0x012711 thru 0x020af0 (0xe3e0 bytes) 000365 1713486168.965517 trunc from 0x03cb4b to 0x023b00 (0x1904a bytes) 000366 1713486168.974068 write 0x02625d thru 0x02729d (0x1041 bytes) 000368 1713486168.974752 write 0x0047fe thru 0x008d16 (0x4519 bytes) 000370 1713486168.977371 write 0x008dac thru 0x017e88 (0xf0dd bytes) 000376 1713486168.980391 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000377 1713486169.026260 trunc from 0x02729e to 0x003f18 (0x23385 bytes) 000378 1713486169.030039 mapwrite 0x027cad thru 0x034aa7 (0xcdfb bytes) 000379 1713486169.053338 mapread 0x02ebd2 thru 0x03398f (0x4dbe bytes) 000380 1713486169.053758 read 0x015e20 thru 0x01eca8 (0x8e89 bytes) 000381 1713486169.056201 mapread 0x006bbc thru 0x00f5b0 (0x89f5 bytes) 000382 1713486169.060970 mapwrite 0x015ba5 thru 0x0234b5 (0xd911 bytes) truncating to largest ever: 0x3a1f8 000383 1713486169.080253 trunc from 0x034aa8 to 0x03a1f7 (0x5750 bytes) 000385 1713486169.084555 mapwrite 0x0299bd thru 0x035592 (0xbbd6 bytes) 000387 1713486169.103365 trunc from 0x03a1f8 to 0x01880b (0x219ec bytes) 000392 1713486169.108413 mapwrite 0x000e5c thru 0x0056bb (0x4860 bytes) 000393 1713486169.127192 MIRROR_RESYNC 2 mirrors 000394 1713486169.217187 mapread 0x0143df thru 0x015208 (0xe2a bytes) 000397 1713486169.219859 read 0x00c1ea thru 0x00d1a0 (0xfb7 bytes) 000399 1713486169.222508 MIRROR_RESYNC 2 mirrors 000403 1713486169.253720 MIRROR_RESYNC 2 mirrors 000404 1713486169.288310 MIRROR_EXTEND to 3 mirrors 000406 1713486169.397144 MIRROR_RESYNC 3 mirrors 000407 1713486169.438002 mapread 0x0061af thru 0x00f85e (0x96b0 bytes) 000408 1713486169.446096 read 0x013c08 thru 0x01880b (0x4c04 bytes) 000410 1713486169.450050 write 0x0251d6 thru 0x026b07 (0x1932 bytes) 000411 1713486169.465416 trunc from 0x026b08 to 0x02c493 (0x598c bytes) 000412 1713486169.474793 write 0x00d6ca thru 0x01a269 (0xcba0 bytes) 000413 1713486169.481442 MIRROR_EXTEND to 4 mirrors 000414 1713486169.610001 trunc from 0x02c494 to 0x02f950 (0x34bd bytes) 000415 1713486169.631109 mapwrite 0x02f723 thru 0x03f6a0 (0xff7e bytes) 000416 1713486169.663361 mapwrite 0x0307e0 thru 0x034ca7 (0x44c8 bytes) 000418 1713486169.679336 mapwrite 0x017fa1 thru 0x01c0e5 (0x4145 bytes) 000419 1713486169.697875 MIRROR_EXTEND to 5 mirrors 000420 1713486169.811996 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000421 1713486169.864481 mapread 0x00b4ce thru 0x010e0c (0x593f bytes) 000423 1713486169.869510 mapread 0x0196ab thru 0x01d685 (0x3fdb bytes) 000424 1713486169.872334 read 0x032a4f thru 0x03df3a (0xb4ec bytes) 000431 1713486169.875391 mapwrite 0x0339dd thru 0x03ffff (0xc623 bytes) 000434 1713486169.911409 mapwrite 0x0114fa thru 0x01be9d (0xa9a4 bytes) 000435 1713486169.929157 mapread 0x009eb7 thru 0x0170ea (0xd234 bytes) 000438 1713486169.932893 read 0x01d002 thru 0x01f84e (0x284d bytes) 000439 1713486169.933282 trunc from 0x040000 to 0x015209 (0x2adf6 bytes) 000441 1713486169.938716 mapread 0x0135c9 thru 0x015209 (0x1c41 bytes) 000443 1713486169.939634 mapread 0x00b684 thru 0x015209 (0x9b86 bytes) 000444 1713486169.940210 trunc from 0x01520a to 0x02e228 (0x1901f bytes) 000446 1713486169.945498 MIRROR_EXTEND to 5 mirrors 000447 1713486170.065674 mapwrite 0x02035f thru 0x02b2e5 (0xaf87 bytes) 000448 1713486170.119995 MIRROR_EXTEND to 6 mirrors truncating to largest ever: 0x3dee2 000450 1713486170.233886 trunc from 0x02e229 to 0x03dee1 (0xfcb9 bytes) 000451 1713486170.249014 mapwrite 0x01096d thru 0x01cb22 (0xc1b6 bytes) 000453 1713486170.274157 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000455 1713486170.317405 mapread 0x0370e1 thru 0x03ab31 (0x3a51 bytes) 000457 1713486170.322643 mapwrite 0x03f1fd thru 0x03ffff (0xe03 bytes) 000460 1713486170.340575 read 0x00da48 thru 0x0130d0 (0x5689 bytes) 000461 1713486170.343487 read 0x009917 thru 0x00a02e (0x718 bytes) 000462 1713486170.346805 trunc from 0x040000 to 0x025c58 (0x1a3a7 bytes) 000463 1713486170.350900 read 0x014362 thru 0x017d16 (0x39b5 bytes) 000464 1713486170.351196 mapwrite 0x032bbc thru 0x03cb22 (0x9f67 bytes) 000465 1713486170.375710 write 0x008323 thru 0x01390e (0xb5ec bytes) 000466 1713486170.376616 write 0x03ac1c thru 0x03ffff (0x53e4 bytes) 000468 1713486170.378942 MIRROR_EXTEND to 6 mirrors 000470 1713486170.512004 read 0x03cf80 thru 0x03ffff (0x3080 bytes) 000472 1713486170.515205 mapwrite 0x004225 thru 0x0099b7 (0x5793 bytes) 000473 1713486170.563720 write 0x031f0d thru 0x03972e (0x7822 bytes) 000474 1713486170.567140 mapread 0x024e0c thru 0x02ec26 (0x9e1b bytes) 000475 1713486170.569968 mapread 0x018804 thru 0x01f0df (0x68dc bytes) 000476 1713486170.572583 mapread 0x02434b thru 0x02ec74 (0xa92a bytes) 000481 1713486170.574147 write 0x024c47 thru 0x02bdc7 (0x7181 bytes) 000485 1713486170.575767 mapwrite 0x03fa11 thru 0x03ffff (0x5ef bytes) 000486 1713486170.591038 read 0x01f150 thru 0x02c7d2 (0xd683 bytes) 000487 1713486170.591542 read 0x03aabc thru 0x03eed0 (0x4415 bytes) 000488 1713486170.591893 mapwrite 0x00339c thru 0x00cf10 (0x9b75 bytes) 000489 1713486170.605728 read 0x0243eb thru 0x032174 (0xdd8a bytes) 000490 1713486170.606250 read 0x034675 thru 0x03c1c4 (0x7b50 bytes) 000491 1713486170.606531 mapread 0x02522c thru 0x034010 (0xede5 bytes) 000492 1713486170.607153 trunc from 0x040000 to 0x0200bc (0x1ff43 bytes) 000495 1713486170.616587 write 0x011faf thru 0x019b9a (0x7bec bytes) truncating to largest ever: 0x3f318 000496 1713486170.619175 trunc from 0x0200bd to 0x03f317 (0x1f25b bytes) 000497 1713486170.622729 read 0x03cb15 thru 0x03f317 (0x2803 bytes) 000498 1713486170.626214 mapread 0x00dbff thru 0x016dbf (0x91c1 bytes) 000499 1713486170.628059 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000501 1713486170.674938 mapread 0x028d9d thru 0x033181 (0xa3e5 bytes) 000504 1713486170.682082 mapread 0x0247fb thru 0x026311 (0x1b17 bytes) 000505 1713486170.682633 write 0x01e1ae thru 0x029bd6 (0xba29 bytes) 000508 1713486170.688120 MIRROR_EXTEND to 6 mirrors 000510 1713486170.805048 mapwrite 0x02c8b8 thru 0x02ccbe (0x407 bytes) 000511 1713486170.850487 read 0x0005f7 thru 0x00f639 (0xf043 bytes) 000512 1713486170.852691 write 0x006961 thru 0x010215 (0x98b5 bytes) 000513 1713486170.854271 read 0x03e36c thru 0x03f317 (0xfac bytes) 000515 1713486170.858068 mapread 0x02c5e4 thru 0x02ee01 (0x281e bytes) 000516 1713486170.858621 mapwrite 0x00c4f3 thru 0x011309 (0x4e17 bytes) 000517 1713486170.873609 write 0x01d8d0 thru 0x022ca6 (0x53d7 bytes) 000518 1713486170.874390 mapread 0x006d6b thru 0x014d0c (0xdfa2 bytes) 000520 1713486170.875324 mapwrite 0x01044f thru 0x01f76f (0xf321 bytes) 000522 1713486170.894272 MIRROR_EXTEND to 7 mirrors 000526 1713486171.011346 MIRROR_RESYNC 7 mirrors 000527 1713486171.226644 trunc from 0x03f318 to 0x000b1d (0x3e7fa bytes) 000528 1713486171.247206 trunc from 0x000b1e to 0x000a8a (0x93 bytes) 000529 1713486171.251974 mapwrite 0x002c10 thru 0x008a72 (0x5e63 bytes) 000531 1713486171.275605 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000532 1713486171.319424 mapwrite 0x03c5be thru 0x03ffff (0x3a42 bytes) 000535 1713486171.342273 write 0x011576 thru 0x01d5ea (0xc075 bytes) 000537 1713486171.348323 MIRROR_RESYNC 6 mirrors 000541 1713486171.501938 read 0x00d8ff thru 0x01051c (0x2c1e bytes) 000542 1713486171.504401 MIRROR_RESYNC 6 mirrors 000543 1713486171.551792 MIRROR_RESYNC 6 mirrors 000545 1713486171.601520 read 0x02d850 thru 0x02e5e5 (0xd96 bytes) 000546 1713486171.603114 trunc from 0x040000 to 0x02464a (0x1b9b5 bytes) 000547 1713486171.621171 write 0x01fe6b thru 0x02a08f (0xa225 bytes) 000548 1713486171.623064 write 0x00ad19 thru 0x019d11 (0xeff9 bytes) 000549 1713486171.625742 trunc from 0x02a090 to 0x01b92b (0xe764 bytes) 000551 1713486171.630142 mapread 0x00c32e thru 0x019709 (0xd3dc bytes) 000552 1713486171.630791 read 0x01b7e7 thru 0x01b92b (0x145 bytes) 000553 1713486171.633540 trunc from 0x01b92c to 0x017386 (0x45a5 bytes) 000557 1713486171.639032 trunc from 0x017387 to 0x02028e (0x8f08 bytes) 000558 1713486171.643278 mapwrite 0x02d9ac thru 0x035738 (0x7d8d bytes) 000562 1713486171.668222 mapread 0x02dc82 thru 0x033ac8 (0x5e47 bytes) 000563 1713486171.668685 write 0x026da9 thru 0x02b017 (0x426f bytes) 000565 1713486171.670893 MIRROR_EXTEND to 7 mirrors 000566 1713486171.777202 trunc from 0x035739 to 0x03183a (0x3efe bytes) 000567 1713486171.798462 mapread 0x012804 thru 0x0154fe (0x2cfb bytes) 000568 1713486171.802676 read 0x0147f7 thru 0x014a3d (0x247 bytes) 000569 1713486171.803006 trunc from 0x03183b to 0x011b19 (0x1fd21 bytes) 000573 1713486171.808811 mapwrite 0x00e998 thru 0x0160c8 (0x7731 bytes) 000574 1713486171.832420 trunc from 0x0160c9 to 0x01210d (0x3fbb bytes) 000575 1713486171.836270 mapread 0x007879 thru 0x01210d (0xa895 bytes) 000576 1713486171.839333 mapwrite 0x0279ff thru 0x02cb09 (0x510b bytes) 000577 1713486171.863828 read 0x0241ad thru 0x02cb09 (0x895d bytes) 000578 1713486171.864292 mapread 0x01bd21 thru 0x02429c (0x857c bytes) 000579 1713486171.868347 read 0x004c1f thru 0x006735 (0x1b17 bytes) 000580 1713486171.868716 mapread 0x01d243 thru 0x024ddf (0x7b9d bytes) 000583 1713486171.869882 mapread 0x00f5c8 thru 0x0181e3 (0x8c1c bytes) 000584 1713486171.872349 MIRROR_EXTEND to 8 mirrors 000590 1713486171.986898 read 0x008ddf thru 0x01393e (0xab60 bytes) 000591 1713486171.992075 MIRROR_EXTEND to 9 mirrors 000593 1713486172.141374 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000595 1713486172.203578 trunc from 0x02cb0a to 0x015b18 (0x16ff1 bytes) 000599 1713486172.225030 write 0x036aef thru 0x03ffff (0x9511 bytes) 000600 1713486172.226089 mapread 0x032615 thru 0x03f3c2 (0xcdae bytes) 000601 1713486172.230481 mapwrite 0x022473 thru 0x025ff4 (0x3b82 bytes) 000603 1713486172.247977 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000604 1713486172.296548 mapread 0x02d4a4 thru 0x03b313 (0xde70 bytes) 000606 1713486172.304000 mapread 0x02972d thru 0x02aa5a (0x132e bytes) 000608 1713486172.308163 MIRROR_RESYNC 7 mirrors 000610 1713486172.504027 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000612 1713486172.571018 trunc from 0x040000 to 0x02b114 (0x14eeb bytes) 000613 1713486172.593015 trunc from 0x02b115 to 0x025c1d (0x54f7 bytes) 000614 1713486172.598429 read 0x0138e9 thru 0x016756 (0x2e6e bytes) 000615 1713486172.602248 MIRROR_RESYNC 6 mirrors 000616 1713486172.781796 mapwrite 0x01c90f thru 0x022e5e (0x6550 bytes) 000617 1713486172.827996 read 0x014483 thru 0x01b4ed (0x706b bytes) 000620 1713486172.830761 mapwrite 0x032dc2 thru 0x039005 (0x6244 bytes) 000621 1713486172.863558 read 0x013b5a thru 0x016f20 (0x33c7 bytes) 000622 1713486172.869019 MIRROR_EXTEND to 7 mirrors 000623 1713486172.982938 mapread 0x0190a0 thru 0x028ee5 (0xfe46 bytes) 000624 1713486172.989899 mapwrite 0x010ac8 thru 0x01d0f1 (0xc62a bytes) 000625 1713486173.040260 mapread 0x028b7d thru 0x032823 (0x9ca7 bytes) 000628 1713486173.046424 MIRROR_EXTEND to 8 mirrors 000629 1713486173.146104 read 0x01d4a9 thru 0x02262b (0x5183 bytes) 000630 1713486173.149074 mapread 0x022719 thru 0x02496e (0x2256 bytes) 000631 1713486173.152973 mapwrite 0x03b777 thru 0x03ba98 (0x322 bytes) 000632 1713486173.189953 mapwrite 0x0102ce thru 0x01374b (0x347e bytes) 000633 1713486173.203750 write 0x0297e9 thru 0x03284f (0x9067 bytes) 000636 1713486173.208179 trunc from 0x03ba99 to 0x012c84 (0x28e14 bytes) 000637 1713486173.213073 mapwrite 0x03f51c thru 0x03ffff (0xae4 bytes) 000639 1713486173.231401 write 0x02d9db thru 0x0365ea (0x8c10 bytes) 000640 1713486173.235649 write 0x036bcf thru 0x03ffff (0x9431 bytes) 000642 1713486173.236456 mapread 0x039626 thru 0x03ffff (0x69da bytes) 000644 1713486173.238193 MIRROR_EXTEND to 9 mirrors 000645 1713486173.358234 write 0x038bda thru 0x03966a (0xa91 bytes) 000646 1713486173.373183 trunc from 0x040000 to 0x039210 (0x6def bytes) 000647 1713486173.381165 read 0x001c56 thru 0x00a6f0 (0x8a9b bytes) 000648 1713486173.383553 write 0x0023cf thru 0x00ebe9 (0xc81b bytes) 000650 1713486173.385614 trunc from 0x039211 to 0x016a21 (0x227ef bytes) 000651 1713486173.389551 mapwrite 0x00629a thru 0x00b897 (0x55fe bytes) 000652 1713486173.404952 mapread 0x01535b thru 0x016a21 (0x16c7 bytes) 000655 1713486173.409681 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000656 1713486173.453232 mapwrite 0x012b2e thru 0x021277 (0xe74a bytes) 000657 1713486173.483180 read 0x01dae9 thru 0x021277 (0x378f bytes) 000658 1713486173.483615 trunc from 0x021278 to 0x012af3 (0xe784 bytes) 000659 1713486173.489121 MIRROR_RESYNC 8 mirrors 000662 1713486173.706199 mapwrite 0x0064c5 thru 0x006eae (0x9ea bytes) 000664 1713486173.750680 read 0x002a34 thru 0x010a1e (0xdfeb bytes) 000665 1713486173.753637 mapread 0x00b09d thru 0x012af3 (0x7a57 bytes) 000666 1713486173.756818 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000667 1713486173.804119 write 0x01d20a thru 0x029f62 (0xcd59 bytes) 000668 1713486173.806100 MIRROR_EXTEND to 8 mirrors 000669 1713486173.919356 write 0x00de99 thru 0x0191e9 (0xb351 bytes) 000670 1713486173.937330 mapread 0x0062af thru 0x011369 (0xb0bb bytes) 000671 1713486173.940287 read 0x027d44 thru 0x029f62 (0x221f bytes) 000672 1713486173.942627 read 0x01af2c thru 0x022135 (0x720a bytes) 000673 1713486173.945653 read 0x00283b thru 0x011974 (0xf13a bytes) 000675 1713486173.946447 write 0x01de76 thru 0x02b644 (0xd7cf bytes) 000678 1713486173.948218 mapread 0x0059cd thru 0x010029 (0xa65d bytes) 000679 1713486173.948996 mapwrite 0x02372d thru 0x02e2eb (0xabbf bytes) 000680 1713486173.981184 trunc from 0x02e2ec to 0x023c2d (0xa6be bytes) 000682 1713486173.985948 trunc from 0x023c2e to 0x036041 (0x12414 bytes) 000683 1713486173.989690 write 0x037cd9 thru 0x03ffff (0x8327 bytes) 000685 1713486173.990686 mapread 0x03a429 thru 0x03ffff (0x5bd7 bytes) 000686 1713486173.992510 MIRROR_EXTEND to 9 mirrors 000687 1713486174.141359 read 0x02afa5 thru 0x037c60 (0xccbc bytes) 000688 1713486174.145960 read 0x03ba4f thru 0x03ffff (0x45b1 bytes) 000690 1713486174.153053 write 0x0118c6 thru 0x01ad49 (0x9484 bytes) 000695 1713486174.179500 read 0x03a463 thru 0x03d7f8 (0x3396 bytes) 000699 1713486174.184995 mapread 0x02d1b4 thru 0x02edeb (0x1c38 bytes) 000701 1713486174.185802 trunc from 0x040000 to 0x028920 (0x176df bytes) 000702 1713486174.194025 MIRROR_EXTEND to 10 mirrors 000703 1713486174.305542 trunc from 0x028921 to 0x01ae2d (0xdaf3 bytes) 000704 1713486174.326507 mapread 0x00e7a4 thru 0x00f308 (0xb65 bytes) 000705 1713486174.329342 write 0x028ba4 thru 0x036cae (0xe10b bytes) 000706 1713486174.330422 mapread 0x01cebc thru 0x02b670 (0xe7b5 bytes) 000707 1713486174.335341 write 0x0197ce thru 0x01ce41 (0x3674 bytes) 000708 1713486174.337232 MIRROR_EXTEND to 11 mirrors 000709 1713486174.489886 mapread 0x00366f thru 0x004ae8 (0x147a bytes) 000711 1713486174.493031 read 0x0098c4 thru 0x00c3e6 (0x2b23 bytes) 000712 1713486174.494549 mapwrite 0x028e57 thru 0x034419 (0xb5c3 bytes) 000713 1713486174.550420 MIRROR_EXTEND to 12 mirrors 000715 1713486174.660607 read 0x008ed9 thru 0x00a602 (0x172a bytes) 000716 1713486174.663271 read 0x033ef9 thru 0x036cae (0x2db6 bytes) 000718 1713486174.668539 mapread 0x00e1f3 thru 0x00e304 (0x112 bytes) 000721 1713486174.672995 mapwrite 0x0330a6 thru 0x03ffff (0xcf5a bytes) 000722 1713486174.717763 mapwrite 0x02cb03 thru 0x0381e6 (0xb6e4 bytes) 000724 1713486174.740959 MIRROR_EXTEND to 13 mirrors 000725 1713486174.900071 read 0x03e544 thru 0x03ffff (0x1abc bytes) 000726 1713486174.905171 MIRROR_EXTEND to 14 mirrors 000727 1713486175.065621 mapread 0x028431 thru 0x032f87 (0xab57 bytes) 000728 1713486175.077667 trunc from 0x040000 to 0x011d7b (0x2e284 bytes) 000730 1713486175.109768 mapwrite 0x02ddfe thru 0x031e34 (0x4037 bytes) 000731 1713486175.153953 mapwrite 0x021fdb thru 0x026610 (0x4636 bytes) 000732 1713486175.189325 trunc from 0x031e35 to 0x03088f (0x15a5 bytes) 000733 1713486175.199040 read 0x01bbda thru 0x01db54 (0x1f7b bytes) 000735 1713486175.201538 MIRROR_EXTEND to 15 mirrors 000737 1713486175.376545 mapread 0x009070 thru 0x018815 (0xf7a6 bytes) 000738 1713486175.394601 mapwrite 0x0155d6 thru 0x01ce57 (0x7882 bytes) 000743 1713486175.467635 write 0x00ff5c thru 0x0110b5 (0x115a bytes) 000744 1713486175.468298 mapread 0x02bcaa thru 0x02f398 (0x36ef bytes) 000745 1713486175.471766 read 0x0050c5 thru 0x00ce22 (0x7d5e bytes) 000746 1713486175.474829 trunc from 0x030890 to 0x01f85c (0x11033 bytes) 000751 1713486175.483483 mapread 0x01d35f thru 0x01f85c (0x24fe bytes) 000752 1713486175.486181 read 0x01d44b thru 0x01e6d9 (0x128f bytes) 000753 1713486175.486564 trunc from 0x01f85d to 0x009dde (0x15a7e bytes) 000754 1713486175.492835 read 0x006d6e thru 0x007e2a (0x10bd bytes) 000755 1713486175.493268 read 0x005efe thru 0x009dde (0x3ee1 bytes) 000756 1713486175.496494 MIRROR_SPLIT mirror 8 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000757 1713486175.574145 MIRROR_EXTEND to 15 mirrors 000759 1713486175.699572 read 0x006587 thru 0x009dde (0x3858 bytes) 000763 1713486175.706176 mapwrite 0x030ce7 thru 0x035677 (0x4991 bytes) 000764 1713486175.756223 read 0x00a683 thru 0x018bda (0xe558 bytes) 000765 1713486175.761645 write 0x0247a3 thru 0x02bcab (0x7509 bytes) 000767 1713486175.765106 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000768 1713486175.825550 read 0x007dd6 thru 0x00db0c (0x5d37 bytes) 000771 1713486175.829048 trunc from 0x035678 to 0x03112d (0x454a bytes) 000772 1713486175.834044 read 0x02e773 thru 0x03112d (0x29bb bytes) 000773 1713486175.839113 trunc from 0x03112e to 0x020023 (0x1110a bytes) 000774 1713486175.844135 write 0x039b15 thru 0x03bc03 (0x20ef bytes) 000775 1713486175.844833 trunc from 0x03bc04 to 0x02aedb (0x10d28 bytes) 000777 1713486175.850141 read 0x00f235 thru 0x018ba4 (0x9970 bytes) 000779 1713486175.850736 mapwrite 0x01a87e thru 0x028b8e (0xe311 bytes) 000780 1713486175.875600 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000781 1713486175.932763 write 0x018507 thru 0x025664 (0xd15e bytes) 000782 1713486175.937384 trunc from 0x02aedc to 0x02936a (0x1b71 bytes) 000783 1713486175.943048 trunc from 0x02936b to 0x01e9f8 (0xa972 bytes) 000784 1713486175.948745 write 0x0316cf thru 0x03f328 (0xdc5a bytes) 000786 1713486175.950047 write 0x01a1b6 thru 0x028cd0 (0xeb1b bytes) 000789 1713486175.953402 mapread 0x00d41f thru 0x00f285 (0x1e67 bytes) 000790 1713486175.957139 mapwrite 0x031775 thru 0x03e1b2 (0xca3e bytes) 000791 1713486175.986216 MIRROR_EXTEND to 14 mirrors 000792 1713486176.137046 write 0x0276bb thru 0x02b1b5 (0x3afb bytes) 000793 1713486176.154743 trunc from 0x03f329 to 0x03ba80 (0x38a8 bytes) 000794 1713486176.159736 mapwrite 0x038c8e thru 0x038d89 (0xfc bytes) 000796 1713486176.177658 read 0x01e3e3 thru 0x020cf4 (0x2912 bytes) 000798 1713486176.180039 mapread 0x02a327 thru 0x0375ba (0xd294 bytes) 000801 1713486176.183526 mapread 0x015425 thru 0x015c39 (0x815 bytes) 000802 1713486176.188622 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000803 1713486176.245299 write 0x01fe58 thru 0x0219aa (0x1b53 bytes) 000804 1713486176.248504 write 0x03bf53 thru 0x03d70c (0x17ba bytes) 000807 1713486176.252763 MIRROR_RESYNC 13 mirrors 000809 1713486176.769420 write 0x00db12 thru 0x01b601 (0xdaf0 bytes) 000810 1713486176.790307 read 0x03c688 thru 0x03d70c (0x1085 bytes) 000812 1713486176.794020 MIRROR_EXTEND to 14 mirrors 000813 1713486176.926129 trunc from 0x03d70d to 0x012bc8 (0x2ab44 bytes) 000816 1713486176.946994 mapwrite 0x002403 thru 0x004d5e (0x295c bytes) 000817 1713486176.969715 mapwrite 0x018c35 thru 0x01e9e3 (0x5daf bytes) 000818 1713486176.996500 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000819 1713486177.044474 mapread 0x016eb5 thru 0x01e9e3 (0x7b2f bytes) 000822 1713486177.050491 write 0x02f0ce thru 0x037934 (0x8867 bytes) 000823 1713486177.051448 mapwrite 0x030b12 thru 0x0356b0 (0x4b9f bytes) 000824 1713486177.069985 write 0x029486 thru 0x02bf22 (0x2a9d bytes) 000827 1713486177.074360 mapread 0x033964 thru 0x033fe5 (0x682 bytes) 000829 1713486177.076784 MIRROR_EXTEND to 14 mirrors 000832 1713486177.212942 MIRROR_RESYNC 14 mirrors 000836 1713486177.576153 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000840 1713486177.678885 read 0x00cff6 thru 0x015869 (0x8874 bytes) 000841 1713486177.684667 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000842 1713486177.790087 mapwrite 0x02426f thru 0x02ee82 (0xac14 bytes) 000843 1713486177.848988 write 0x03b70b thru 0x03bdbe (0x6b4 bytes) 000844 1713486177.849568 mapread 0x00f48b thru 0x01bddd (0xc953 bytes) 000847 1713486177.854676 write 0x0037a6 thru 0x00c60e (0x8e69 bytes) 000849 1713486177.857812 trunc from 0x03bdbf to 0x03e4a3 (0x26e5 bytes) 000850 1713486177.863327 trunc from 0x03e4a4 to 0x0321ac (0xc2f7 bytes) 000853 1713486177.868232 trunc from 0x0321ad to 0x0021cb (0x2ffe1 bytes) 000854 1713486177.873143 read 0x0007f5 thru 0x0021cb (0x19d7 bytes) 000855 1713486177.874570 mapwrite 0x01c253 thru 0x02029f (0x404d bytes) 000857 1713486177.896171 mapread 0x0165b9 thru 0x018a99 (0x24e1 bytes) 000858 1713486177.896722 mapread 0x010a1f thru 0x012c91 (0x2273 bytes) 000859 1713486177.899582 mapread 0x01e31e thru 0x02029f (0x1f82 bytes) 000862 1713486177.900433 read 0x00eb85 thru 0x01b080 (0xc4fc bytes) 000863 1713486177.900747 write 0x02edaa thru 0x03a4de (0xb735 bytes) 000864 1713486177.903052 MIRROR_EXTEND to 13 mirrors 000865 1713486178.025052 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000867 1713486178.107026 mapwrite 0x00d99d thru 0x019780 (0xbde4 bytes) 000868 1713486178.162570 mapread 0x033382 thru 0x03521e (0x1e9d bytes) 000869 1713486178.165910 mapwrite 0x035471 thru 0x03ffff (0xab8f bytes) 000870 1713486178.197871 mapwrite 0x00b3ff thru 0x00ece4 (0x38e6 bytes) 000872 1713486178.212434 mapwrite 0x000b9f thru 0x006614 (0x5a76 bytes) 000875 1713486178.234500 write 0x03ec69 thru 0x03f646 (0x9de bytes) 000876 1713486178.235303 trunc from 0x040000 to 0x01f170 (0x20e8f bytes) 000878 1713486178.241276 write 0x01ceea thru 0x026669 (0x9780 bytes) 000879 1713486178.242004 mapwrite 0x016c28 thru 0x01cb64 (0x5f3d bytes) 000880 1713486178.259590 write 0x007165 thru 0x007b0e (0x9aa bytes) 000882 1713486178.260434 trunc from 0x02666a to 0x03bc7c (0x15613 bytes) 000884 1713486178.265212 mapread 0x035095 thru 0x03bc7c (0x6be8 bytes) 000885 1713486178.269668 MIRROR_EXTEND to 13 mirrors 000887 1713486178.433111 trunc from 0x03bc7d to 0x03e01d (0x23a1 bytes) 000889 1713486178.459114 MIRROR_RESYNC 13 mirrors 000891 1713486178.874876 read 0x03cacd thru 0x03e01d (0x1551 bytes) 000892 1713486178.876697 mapread 0x023d19 thru 0x0261e4 (0x24cc bytes) 000893 1713486178.880892 write 0x0020fb thru 0x003d80 (0x1c86 bytes) 000894 1713486178.899042 read 0x02d706 thru 0x03943a (0xbd35 bytes) 000895 1713486178.904080 trunc from 0x03e01e to 0x03cc12 (0x140b bytes) 000897 1713486178.911319 trunc from 0x03cc13 to 0x02b419 (0x117f9 bytes) 000899 1713486178.919019 mapwrite 0x03cc6c thru 0x03ffff (0x3394 bytes) 000901 1713486178.956121 MIRROR_RESYNC 13 mirrors 000902 1713486179.319672 mapread 0x03066f thru 0x039c68 (0x95fa bytes) 000904 1713486179.325129 read 0x011da3 thru 0x018cfe (0x6f5c bytes) 000905 1713486179.327143 mapwrite 0x0157be thru 0x018a8f (0x32d2 bytes) 000907 1713486179.375549 mapwrite 0x001b25 thru 0x009e46 (0x8322 bytes) 000909 1713486179.397688 write 0x03866f thru 0x03a9f3 (0x2385 bytes) 000910 1713486179.402032 mapwrite 0x037cbb thru 0x03ea5a (0x6da0 bytes) 000911 1713486179.418759 mapwrite 0x0324e7 thru 0x037629 (0x5143 bytes) 000912 1713486179.436414 read 0x01c57d thru 0x028f8c (0xca10 bytes) 000913 1713486179.440163 write 0x0022ac thru 0x002846 (0x59b bytes) 000919 1713486179.441985 mapread 0x021c6d thru 0x0318c0 (0xfc54 bytes) 000920 1713486179.442660 trunc from 0x040000 to 0x02de4d (0x121b2 bytes) 000923 1713486179.449211 mapwrite 0x03ec12 thru 0x03ffff (0x13ee bytes) 000924 1713486179.476590 MIRROR_RESYNC 13 mirrors 000925 1713486179.817440 write 0x028dc3 thru 0x036340 (0xd57e bytes) 000926 1713486179.834356 mapread 0x026cec thru 0x026f77 (0x28c bytes) 000927 1713486179.837512 read 0x01d407 thru 0x024c36 (0x7830 bytes) 000930 1713486179.839993 mapwrite 0x03a5f1 thru 0x03ffff (0x5a0f bytes) 000931 1713486179.864359 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000932 1713486179.919589 mapwrite 0x0314c8 thru 0x03bbbe (0xa6f7 bytes) 000935 1713486179.941650 MIRROR_RESYNC 12 mirrors 000938 1713486180.244210 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000940 1713486180.332501 mapread 0x029156 thru 0x031024 (0x7ecf bytes) 000941 1713486180.338493 MIRROR_RESYNC 11 mirrors 000942 1713486180.416390 mapread 0x01da3f thru 0x01f72e (0x1cf0 bytes) 000943 1713486180.424508 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000945 1713486180.508361 write 0x0179a8 thru 0x02539c (0xd9f5 bytes) 000953 1713486180.526328 mapwrite 0x0330e2 thru 0x038727 (0x5646 bytes) 000954 1713486180.545211 trunc from 0x040000 to 0x020323 (0x1fcdc bytes) 000955 1713486180.550534 mapread 0x01a83b thru 0x01aa9f (0x265 bytes) 000957 1713486180.551420 mapwrite 0x009ad0 thru 0x009c19 (0x14a bytes) 000958 1713486180.568410 write 0x01313d thru 0x013595 (0x459 bytes) 000962 1713486180.571740 read 0x01edba thru 0x020323 (0x156a bytes) 000963 1713486180.572054 read 0x000a72 thru 0x002a21 (0x1fb0 bytes) 000964 1713486180.575697 MIRROR_EXTEND to 11 mirrors 000965 1713486180.684010 read 0x0047d8 thru 0x004fa4 (0x7cd bytes) 000966 1713486180.686293 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000967 1713486180.755018 MIRROR_EXTEND to 11 mirrors 000974 1713486180.888362 trunc from 0x020324 to 0x0087d6 (0x17b4d bytes) 000975 1713486180.914865 write 0x039a26 thru 0x03ffff (0x65da bytes) 000976 1713486180.915727 mapread 0x0358c9 thru 0x03e78a (0x8ec2 bytes) 000978 1713486180.921005 mapwrite 0x005358 thru 0x012941 (0xd5ea bytes) 000980 1713486180.950531 write 0x03a9ad thru 0x03bbff (0x1253 bytes) 000981 1713486180.951248 mapread 0x02d28f thru 0x02f0b5 (0x1e27 bytes) 000983 1713486180.954971 mapwrite 0x027a82 thru 0x02a00b (0x258a bytes) 000985 1713486180.974696 mapread 0x030a6e thru 0x0314aa (0xa3d bytes) 000987 1713486180.975545 trunc from 0x040000 to 0x00b1d0 (0x34e2f bytes) 000989 1713486180.983172 mapwrite 0x01cb93 thru 0x02c55d (0xf9cb bytes) 000990 1713486181.012577 mapwrite 0x038077 thru 0x038923 (0x8ad bytes) 000993 1713486181.034250 trunc from 0x038924 to 0x02e16d (0xa7b6 bytes) 000994 1713486181.038229 mapread 0x010bbb thru 0x0171af (0x65f5 bytes) 000995 1713486181.042646 MIRROR_RESYNC 11 mirrors 000996 1713486181.420432 mapwrite 0x023bca thru 0x030e18 (0xd24f bytes) 000997 1713486181.475375 write 0x008e01 thru 0x010424 (0x7624 bytes) 000998 1713486181.480278 MIRROR_EXTEND to 12 mirrors All operations completed A-OK! PASS 70a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 20:23:04 (1713486184) Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3079744 .. Extending file size to 2767328 .. Extending file size to 7391936 .. Extending file size to 5119424 .. Extending file size to 8293120 .. Extending file size to 7817920 .. Extending file size to 8506304 .. Extending file size to 9148288 .. Extending file size to 8528320 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9266880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8632000 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8818112 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8942880 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 2117 2118 2119 2121 2122 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) 192.168.204.141@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg441-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:24:09 (1713486249) Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2753440 .. Extending file size to 2648160 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5125440 .. Extending file size to 8771552 .. failed Extending file size to 5572768 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9100320 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8302784 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8912224 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8965856 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8981312 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 5450 5451 5453 5454 5456 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) 192.168.204.141@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg441-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:25:14 (1713486314) Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4608 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4608 2203904 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0351546 s, 59.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0673646 s, 31.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:25:20 (1713486320) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0242247 s, 86.6 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 20:25:23 (1713486323) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x753:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x753:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x754:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x74e:0x0] } PASS 203 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:25:25 (1713486325) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.024273 s, 86.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x755:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x74f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x750:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:25:29 (1713486329) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0298178 s, 70.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x756:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x757:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x751:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x756:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x757:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x751:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x752:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:25:32 (1713486332) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0256585 s, 81.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x758:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x753:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x758:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x753:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x754:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:25:56 (1713486356) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0268077 s, 39.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x759:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x756:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x755:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x759:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x756:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x755:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x75a:0x0] } PASS 204d (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:26:39 (1713486399) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x757:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x758:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.026455 s, 79.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x757:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x758:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x75b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x757:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x758:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x75b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:27:24 (1713486444) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. Creating new pool oleg441-server: Pool lustre.test_204f created Adding targets to pool oleg441-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg441-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0229669 s, 91.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x759:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x75c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x759:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x75c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg441-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg441-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg441-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (50s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 20:28:15 (1713486495) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:28:18 (1713486498) oleg441-server: Pool lustre.test_206 created Adding targets to pool oleg441-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg441-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg441-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg441-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg441-server: Pool lustre.test_206 destroyed PASS 206 (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:28:32 (1713486512) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162822 s, 64.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00999659 s, 105 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x9a6bbf09 Mirror 3: 0x9a6bbf09 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:28:36 (1713486516) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:28:38 (1713486518) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:28:40 (1713486520) Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0151351 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1510000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1980000 Read this (4 bytes): BBBB 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:28:55 (1713486535) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00267778 s, 3.1 MB/s the very first read read with warm cache PASS 209b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 20:28:58 (1713486538) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00952844 s, 110 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x44f lmm_fid: [0x200000404:0x44f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x44f lmm_fid: [0x200000404:0x44f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x763:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835023 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-18 20:28:58.000000000 -0400 Modify: 2024-04-18 20:28:58.000000000 -0400 Change: 2024-04-18 20:28:59.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:29:01 (1713486541) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 769 sec ============ 20:29:02 (1713486542) === sanity-flr: start cleanup 20:29:03 (1713486543) === === sanity-flr: finish cleanup 20:29:03 (1713486543) ===