-----============= acceptance-small: sanity-flr ============----- Tue Apr 16 15:40:46 EDT 2024 excepting tests: 6 201 44c === sanity-flr: start setup 15:40:49 (1713296449) === oleg146-client.virtnet: executing check_config_client /mnt/lustre oleg146-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg146-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012bf34800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012bf34800.idle_timeout=debug disable quota as required oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:40:56 (1713296456) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7500] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 15:40:57 (1713296457) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:41:01 (1713296461) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:41:04 (1713296464) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 15:41:06 (1713296466) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:41:11 (1713296471) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:41:13 (1713296473) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 15:41:16 (1713296476) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012bf34800.stats=clear osc.lustre-OST0001-osc-ffff88012bf34800.stats=clear PASS 0g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:41:19 (1713296479) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:41:22 (1713296482) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:41:25 (1713296485) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 15:41:27 (1713296487) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 15:41:30 (1713296490) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:41:33 (1713296493) PASS 3 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:41:36 (1713296496) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:41:39 (1713296499) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 15:41:43 (1713296503) PASS 7 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:41:46 (1713296506) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.141613 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.02492 s, 42.1 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:41:49 (1713296509) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0446294 s, 141 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:41:53 (1713296513) Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012bf34800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012bf34800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012bf34800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012bf34800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:42:13 (1713296533) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.127893 s, 65.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012bf34800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012bf34800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:42:23 (1713296543) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012bf34800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012bf34800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012bf34800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012bf34800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:42:43 (1713296563) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.833613 s, 62.9 MB/s -rw-r--r-- 1 root root 52428800 Apr 16 15:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.208s user 0m0.000s sys 0m0.122s Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012bf34800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012bf34800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.215s user 0m0.000s sys 0m0.133s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012bf34800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012bf34800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.212s user 0m0.001s sys 0m0.109s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg146-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012bf34800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012bf34800.ost_server_uuid in FULL state after 0 sec PASS 33b (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:43:09 (1713296589) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:43:11 (1713296591) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:43:14 (1713296594) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 15:43:16 (1713296596) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 15:43:19 (1713296599) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.124631 s, 101 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:43:23 (1713296603) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00752869 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00769827 s, 136 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:43:26 (1713296606) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:43:30 (1713296610) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000612149 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000523014 s, 2.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000607384 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000501736 s, 2.0 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 15:43:33 (1713296613) create test files with size 15748502 .. ab4e6ccace7e265ca8eb654f0b1c5feb - 54355b177e4d9b8912c2b22eadc2c5a9 - 30e12d4f55173f895dac24f5e423af63 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre PASS 37 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 15:43:39 (1713296619) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:43:43 (1713296623) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:43:46 (1713296626) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132591 s, 79.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101702 s, 103 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 15:43:50 (1713296630) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0198892 s, 105 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0808725 s, 51.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0331669 s, 63.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 15:43:54 (1713296634) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 15:43:56 (1713296636) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:43:59 (1713296639) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00917661 s, 114 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 15:44:02 (1713296642) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0791779 s, 39.7 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:44:06 (1713296646) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4c:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:44:10 (1713296650) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0711738 s, 147 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:44:14 (1713296654) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800aa428800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0732028 s, 143 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800aac6b800.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:44:19 (1713296659) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 15:44:28 (1713296668) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 15:44:35 (1713296675) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 15:44:38 (1713296678) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0677495 s, 46.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.057965 s, 54.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 1edda188d401004e6a31b078ba3304f7 - mirror 1 checksum: 0be4f0c045a36161e67bb04e8098929c - mirror 2 checksum: 1edda188d401004e6a31b078ba3304f7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 15:44:38 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 15:44:38 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1edda188d401004e6a31b078ba3304f7 - mirror 1 checksum: 0be4f0c045a36161e67bb04e8098929c - mirror 2 checksum: 0be4f0c045a36161e67bb04e8098929c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 15:44:38 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 15:44:38 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:44:41 (1713296681) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00957541 s, 110 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:44:45 (1713296685) Failing mds1 on oleg146-server Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 15:44:52 (1713296692) shut down Failover mds1 to oleg146-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 15:45:05 (1713296705) targets are mounted 15:45:05 (1713296705) facet_failover done oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111895 s, 93.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 50A (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:45:14 (1713296714) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00992531 s, 2.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00438499 s, 4.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 15:45:18 (1713296718) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00659161 s, 3.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00222637 s, 9.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:45:21 (1713296721) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00225578 s, 7.3 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 15:45:24 (1713296724) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00877876 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:45:28 (1713296728) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00139375 s, 2.9 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0070228 s, 583 kB/s PASS 60a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:45:31 (1713296731) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:45:37 (1713296737) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:45:56 (1713296756) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:46:14 (1713296774) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:46:33 (1713296793) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 15:46:37 (1713296797) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 30480 mirror split pid 30481 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 30480 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 30481 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 15:47:40 (1713296860) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6860 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x1156a 000001 1713296860.699144 fallocate 0x007785 thru 0x011569 (0x9de5 bytes) skipping zero size read skipping zero length punch hole 000005 1713296860.702503 mapwrite 0x00dfaf thru 0x01111c (0x316e bytes) fallocating to largest ever: 0x13094 000006 1713296860.725731 fallocate 0x00bf38 thru 0x013093 (0x715c bytes) 000007 1713296860.727373 mapread 0x00ad99 thru 0x013093 (0x82fb bytes) 000008 1713296860.729558 write 0x00a8ca thru 0x011853 (0x6f8a bytes) 000010 1713296860.730764 mapread 0x0125b7 thru 0x013093 (0xadd bytes) 000011 1713296860.731203 write 0x03133a thru 0x03ffff (0xecc6 bytes) fallocating to largest ever: 0x40000 000012 1713296860.732013 fallocate 0x03fbf6 thru 0x03ffff (0x40a bytes) 000013 1713296860.734013 punch from 0x030974 to 0x03f6de (0xed6b bytes) 000014 1713296860.738347 punch from 0x01d16c to 0x01df4c (0xde1 bytes) 000015 1713296860.741109 MIRROR_EXTEND to 2 mirrors 000016 1713296860.842628 trunc from 0x040000 to 0x01344f (0x2cbb0 bytes) 000017 1713296860.855612 mapread 0x00f9f3 thru 0x01344f (0x3a5d bytes) 000018 1713296860.860009 mapread 0x0080cc thru 0x01344f (0xb384 bytes) 000019 1713296860.862107 fallocate 0x015645 thru 0x023d0f (0xe6cb bytes) 000020 1713296860.863604 read 0x010671 thru 0x01344f (0x2ddf bytes) 000022 1713296860.864322 read 0x00e946 thru 0x01344f (0x4b0a bytes) 000023 1713296860.864732 fallocate 0x031ba7 thru 0x03b4dd (0x9937 bytes) 000024 1713296860.866246 read 0x008f7f thru 0x01047c (0x74fe bytes) 000025 1713296860.866635 trunc from 0x013450 to 0x0099af (0x9aa0 bytes) 000027 1713296860.871585 mapwrite 0x006f2f thru 0x013c33 (0xcd05 bytes) 000029 1713296860.884989 read 0x013a29 thru 0x013c33 (0x20b bytes) 000030 1713296860.885321 write 0x030f3f thru 0x0349ce (0x3a90 bytes) 000031 1713296860.885975 mapwrite 0x030758 thru 0x03ef61 (0xe80a bytes) 000032 1713296860.901034 write 0x003a2d thru 0x006bfe (0x31d2 bytes) 000033 1713296860.901918 write 0x019ddd thru 0x01fdea (0x600e bytes) 000034 1713296860.902547 punch from 0x01e3c8 to 0x024315 (0x5f4e bytes) 000035 1713296860.908124 MIRROR_RESYNC 2 mirrors 000036 1713296861.014276 MIRROR_EXTEND to 3 mirrors 000037 1713296861.134628 punch from 0x00d215 to 0x0112cf (0x40bb bytes) 000038 1713296861.143442 write 0x02bf5d thru 0x03ab17 (0xebbb bytes) 000040 1713296861.147826 mapwrite 0x0198ba thru 0x028d80 (0xf4c7 bytes) 000042 1713296861.159299 read 0x00403f thru 0x004d7c (0xd3e bytes) 000043 1713296861.160832 read 0x03b9c6 thru 0x03ef61 (0x359c bytes) 000044 1713296861.163152 trunc from 0x03ef62 to 0x01b603 (0x2395e bytes) 000045 1713296861.167944 trunc from 0x01b604 to 0x0144ff (0x7104 bytes) 000046 1713296861.171407 punch from 0x00f044 to 0x0144ff (0x54bc bytes) 000047 1713296861.174159 MIRROR_RESYNC 3 mirrors 000048 1713296861.270346 mapwrite 0x017bdf thru 0x020ba1 (0x8fc3 bytes) 000049 1713296861.291971 trunc from 0x020ba2 to 0x00a3ee (0x167b3 bytes) 000050 1713296861.296341 mapread 0x000d72 thru 0x005ca0 (0x4f2f bytes) 000051 1713296861.299765 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000052 1713296861.346507 mapread 0x005c67 thru 0x007624 (0x19be bytes) 000054 1713296861.352071 mapwrite 0x0245a4 thru 0x02d45b (0x8eb8 bytes) 000055 1713296861.367225 read 0x00b5b8 thru 0x018b4a (0xd593 bytes) 000056 1713296861.369802 read 0x007e1f thru 0x012ccc (0xaeae bytes) 000057 1713296861.370275 punch from 0x02bb63 to 0x02d45b (0x18f9 bytes) 000058 1713296861.372060 write 0x0343aa thru 0x03fb11 (0xb768 bytes) 000060 1713296861.373375 punch from 0x00a3d0 to 0x00ae75 (0xaa6 bytes) 000061 1713296861.374606 fallocate 0x01a6cf thru 0x02a2da (0xfc0c bytes) 000062 1713296861.376055 read 0x02c6f9 thru 0x031fed (0x58f5 bytes) 000064 1713296861.379780 MIRROR_EXTEND to 3 mirrors 000065 1713296861.500321 write 0x02d673 thru 0x035f9c (0x892a bytes) 000066 1713296861.511148 mapread 0x032739 thru 0x0375e3 (0x4eab bytes) 000067 1713296861.513680 read 0x037ca9 thru 0x03fb11 (0x7e69 bytes) 000070 1713296861.514536 fallocate 0x03b0ca thru 0x03ffff (0x4f36 bytes) 000071 1713296861.516022 write 0x01c3d2 thru 0x0234d8 (0x7107 bytes) 000072 1713296861.518306 mapwrite 0x02b723 thru 0x034f06 (0x97e4 bytes) 000073 1713296861.527930 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000075 1713296861.571265 fallocate 0x00be19 thru 0x01a508 (0xe6f0 bytes) 000076 1713296861.572955 mapread 0x026951 thru 0x0333cb (0xca7b bytes) 000077 1713296861.579206 mapread 0x0009c8 thru 0x00cd7b (0xc3b4 bytes) 000078 1713296861.583554 mapread 0x03d4b5 thru 0x03ffff (0x2b4b bytes) 000079 1713296861.585514 mapwrite 0x023b1e thru 0x0261eb (0x26ce bytes) 000080 1713296861.589428 mapread 0x003f85 thru 0x011896 (0xd912 bytes) 000081 1713296861.590015 read 0x029840 thru 0x03923d (0xf9fe bytes) 000083 1713296861.590616 punch from 0x01f017 to 0x026e3d (0x7e27 bytes) 000085 1713296861.592751 punch from 0x023c73 to 0x027144 (0x34d2 bytes) 000086 1713296861.594014 punch from 0x02261b to 0x02650e (0x3ef4 bytes) 000087 1713296861.595154 write 0x00cdb8 thru 0x012d98 (0x5fe1 bytes) 000088 1713296861.597162 mapread 0x0251b1 thru 0x025202 (0x52 bytes) 000091 1713296861.599998 write 0x00dab9 thru 0x012b6b (0x50b3 bytes) 000092 1713296861.601557 MIRROR_EXTEND to 3 mirrors 000093 1713296861.717434 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000094 1713296861.771422 mapread 0x016181 thru 0x021b5e (0xb9de bytes) 000095 1713296861.778707 MIRROR_EXTEND to 3 mirrors 000096 1713296861.887307 mapread 0x002e41 thru 0x01109f (0xe25f bytes) 000098 1713296861.898727 write 0x03da1f thru 0x03e717 (0xcf9 bytes) 000099 1713296861.909783 mapread 0x0300a9 thru 0x033ecf (0x3e27 bytes) 000100 1713296861.912286 write 0x0017d4 thru 0x00c2fb (0xab28 bytes) 000102 1713296861.915489 write 0x03ccc9 thru 0x03e405 (0x173d bytes) 000103 1713296861.916241 write 0x0253a3 thru 0x02df5e (0x8bbc bytes) 000104 1713296861.918919 trunc from 0x040000 to 0x03f8be (0x741 bytes) 000105 1713296861.927670 read 0x012e63 thru 0x015c9f (0x2e3d bytes) 000106 1713296861.929281 punch from 0x02f2bb to 0x034223 (0x4f69 bytes) 000107 1713296861.930644 fallocate 0x01eb19 thru 0x020a97 (0x1f7f bytes) 000108 1713296861.932170 mapwrite 0x02645a thru 0x0269a9 (0x550 bytes) 000110 1713296861.938297 punch from 0x01f45b to 0x028775 (0x931b bytes) 000111 1713296861.940881 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000112 1713296861.987993 trunc from 0x03f8bf to 0x029d36 (0x15b88 bytes) 000113 1713296861.992819 write 0x034420 thru 0x03e9cb (0xa5ac bytes) 000114 1713296861.993669 read 0x007c08 thru 0x011dbb (0xa1b4 bytes) 000115 1713296861.995928 trunc from 0x03e9cc to 0x0030b7 (0x3b914 bytes) 000116 1713296862.000009 fallocate 0x001d9c thru 0x010193 (0xe3f8 bytes) 000117 1713296862.001367 mapwrite 0x012c95 thru 0x01862a (0x5996 bytes) 000118 1713296862.014525 write 0x0357d9 thru 0x035aae (0x2d6 bytes) 000119 1713296862.016604 MIRROR_EXTEND to 3 mirrors 000120 1713296862.128367 trunc from 0x035aaf to 0x00fef9 (0x25bb5 bytes) 000121 1713296862.137664 fallocate 0x03ac79 thru 0x03ffff (0x5387 bytes) 000122 1713296862.139126 punch from 0x01ea31 to 0x02c967 (0xdf37 bytes) 000123 1713296862.140441 fallocate 0x005150 thru 0x00ab0c (0x59bd bytes) 000126 1713296862.142300 mapread 0x000f82 thru 0x005a0b (0x4a8a bytes) 000127 1713296862.147789 punch from 0x00b149 to 0x0185e4 (0xd49c bytes) 000128 1713296862.149194 read 0x034904 thru 0x0355da (0xcd7 bytes) 000129 1713296862.150703 write 0x026476 thru 0x02a4d3 (0x405e bytes) 000131 1713296862.153147 mapread 0x030024 thru 0x037721 (0x76fe bytes) 000132 1713296862.155492 read 0x02c41b thru 0x03709a (0xac80 bytes) 000133 1713296862.157717 read 0x009452 thru 0x012d56 (0x9905 bytes) 000134 1713296862.158014 write 0x02a13b thru 0x035c02 (0xbac8 bytes) 000136 1713296862.158856 write 0x0279c3 thru 0x0376ff (0xfd3d bytes) 000137 1713296862.159567 punch from 0x003aa2 to 0x00eb6b (0xb0ca bytes) 000138 1713296862.161162 fallocate 0x0281b4 thru 0x03140f (0x925c bytes) 000139 1713296862.164952 trunc from 0x040000 to 0x039bd1 (0x642e bytes) 000141 1713296862.169521 punch from 0x0272e4 to 0x02767b (0x398 bytes) 000143 1713296862.173030 fallocate 0x03f489 thru 0x03ffff (0xb77 bytes) 000144 1713296862.174378 fallocate 0x0098fe thru 0x018584 (0xec87 bytes) 000147 1713296862.176038 read 0x01fddf thru 0x02a305 (0xa527 bytes) 000149 1713296862.178567 mapwrite 0x005a16 thru 0x00b074 (0x565f bytes) 000150 1713296862.183938 read 0x005a45 thru 0x00b66e (0x5c2a bytes) 000151 1713296862.184265 trunc from 0x040000 to 0x03ad62 (0x529d bytes) 000153 1713296862.187817 write 0x02a55c thru 0x03a31e (0xfdc3 bytes) 000154 1713296862.188605 read 0x032b07 thru 0x038685 (0x5b7f bytes) 000155 1713296862.188939 mapread 0x025c1d thru 0x0307bf (0xaba3 bytes) 000156 1713296862.190463 MIRROR_EXTEND to 4 mirrors 000157 1713296862.296055 mapread 0x00bc58 thru 0x01110e (0x54b7 bytes) 000158 1713296862.300718 write 0x038f9f thru 0x03ffff (0x7061 bytes) 000159 1713296862.310550 write 0x039735 thru 0x03ffff (0x68cb bytes) 000160 1713296862.311215 write 0x01811a thru 0x022918 (0xa7ff bytes) 000161 1713296862.314785 trunc from 0x040000 to 0x00eadf (0x31520 bytes) 000162 1713296862.323053 read 0x006b19 thru 0x00eadf (0x7fc7 bytes) 000163 1713296862.325306 mapread 0x0033b0 thru 0x0067d7 (0x3428 bytes) 000165 1713296862.327861 fallocate 0x0043b2 thru 0x01042e (0xc07d bytes) 000167 1713296862.329413 trunc from 0x01042f to 0x007005 (0x9429 bytes) 000170 1713296862.333783 fallocate 0x023bd3 thru 0x026cf0 (0x311e bytes) 000171 1713296862.335363 fallocate 0x00b465 thru 0x010980 (0x551c bytes) 000172 1713296862.336636 mapread 0x01863c thru 0x024a62 (0xc427 bytes) 000173 1713296862.341133 mapread 0x025dca thru 0x026cf0 (0xf27 bytes) 000175 1713296862.341867 fallocate 0x011989 thru 0x01e415 (0xca8d bytes) 000176 1713296862.343138 mapwrite 0x034daf thru 0x03e4fe (0x9750 bytes) 000177 1713296862.355941 trunc from 0x03e4ff to 0x039cad (0x4851 bytes) 000178 1713296862.360536 MIRROR_RESYNC 4 mirrors 000179 1713296862.494440 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000180 1713296862.546394 mapread 0x034d2c thru 0x039cad (0x4f82 bytes) 000181 1713296862.551032 trunc from 0x039cae to 0x0223d6 (0x178d7 bytes) 000183 1713296862.565538 mapwrite 0x013cd1 thru 0x0161a4 (0x24d4 bytes) 000184 1713296862.574197 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000185 1713296862.617280 MIRROR_EXTEND to 3 mirrors 000186 1713296862.728417 mapread 0x00f9ca thru 0x0196a5 (0x9cdc bytes) 000187 1713296862.738423 read 0x00e594 thru 0x0110df (0x2b4c bytes) 000188 1713296862.740361 read 0x00d62a thru 0x0160c2 (0x8a99 bytes) 000190 1713296862.745684 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000192 1713296862.794131 MIRROR_EXTEND to 3 mirrors 000193 1713296862.904836 fallocate 0x038b53 thru 0x03fcca (0x7178 bytes) 000194 1713296862.913636 read 0x016917 thru 0x01c8ac (0x5f96 bytes) 000196 1713296862.916029 fallocate 0x02c0e0 thru 0x037882 (0xb7a3 bytes) 000197 1713296862.917534 read 0x02efd9 thru 0x0345c8 (0x55f0 bytes) 000198 1713296862.920634 trunc from 0x037883 to 0x02e364 (0x951e bytes) 000199 1713296862.929401 mapread 0x00be44 thru 0x012b0c (0x6cc9 bytes) 000200 1713296862.929925 write 0x010ec9 thru 0x01874e (0x7886 bytes) 000201 1713296862.930609 write 0x024e9a thru 0x02cd0e (0x7e75 bytes) 000202 1713296862.931232 write 0x02463a thru 0x029484 (0x4e4b bytes) 000204 1713296862.931964 read 0x008bcf thru 0x0133d3 (0xa805 bytes) 000205 1713296862.933509 MIRROR_RESYNC 3 mirrors 000206 1713296863.069002 read 0x001276 thru 0x005406 (0x4191 bytes) 000207 1713296863.072170 MIRROR_RESYNC 3 mirrors 000208 1713296863.107050 mapwrite 0x014561 thru 0x016d08 (0x27a8 bytes) 000210 1713296863.137774 MIRROR_EXTEND to 4 mirrors 000212 1713296863.256344 mapread 0x029bae thru 0x02b4f8 (0x194b bytes) 000213 1713296863.261648 MIRROR_RESYNC 4 mirrors 000214 1713296863.393907 mapwrite 0x005f67 thru 0x00f8cd (0x9967 bytes) 000215 1713296863.422370 mapread 0x00b87b thru 0x010da1 (0x5527 bytes) 000216 1713296863.422890 read 0x027755 thru 0x02e364 (0x6c10 bytes) 000217 1713296863.425089 fallocate 0x0302c7 thru 0x03dff1 (0xdd2b bytes) 000218 1713296863.426771 punch from 0x016928 to 0x01d6b5 (0x6d8e bytes) 000219 1713296863.429202 MIRROR_EXTEND to 5 mirrors 000220 1713296863.529403 MIRROR_RESYNC 5 mirrors 000222 1713296863.680200 fallocate 0x03c2ba thru 0x03ffff (0x3d46 bytes) 000223 1713296863.687846 write 0x023bf8 thru 0x024165 (0x56e bytes) 000224 1713296863.690484 punch from 0x020e63 to 0x02e364 (0xd502 bytes) 000225 1713296863.693462 mapwrite 0x0001f8 thru 0x00b9f8 (0xb801 bytes) 000226 1713296863.702428 read 0x001228 thru 0x00226d (0x1046 bytes) 000227 1713296863.702732 mapwrite 0x002f56 thru 0x00b37b (0x8426 bytes) 000228 1713296863.707857 punch from 0x01ef35 to 0x023bce (0x4c9a bytes) 000230 1713296863.709697 write 0x03f24a thru 0x03ffff (0xdb6 bytes) 000232 1713296863.710806 read 0x035a2a thru 0x03ffa4 (0xa57b bytes) 000233 1713296863.714754 MIRROR_RESYNC 5 mirrors 000236 1713296863.895656 trunc from 0x040000 to 0x025672 (0x1a98d bytes) 000237 1713296863.908929 read 0x018e03 thru 0x01939b (0x599 bytes) 000238 1713296863.910690 punch from 0x013c2f to 0x023467 (0xf839 bytes) 000239 1713296863.912237 read 0x000fd0 thru 0x009138 (0x8169 bytes) 000240 1713296863.914871 mapread 0x00fd42 thru 0x01f139 (0xf3f8 bytes) 000241 1713296863.915502 punch from 0x01f4a7 to 0x025672 (0x61cc bytes) 000242 1713296863.916770 read 0x0189f2 thru 0x0242ad (0xb8bc bytes) 000243 1713296863.918848 mapwrite 0x02ff2d thru 0x0334c0 (0x3594 bytes) 000244 1713296863.930859 punch from 0x00a5c1 to 0x012a5f (0x849f bytes) 000246 1713296863.932648 fallocate 0x02c2b5 thru 0x0389d7 (0xc723 bytes) 000247 1713296863.933970 trunc from 0x0334c1 to 0x03e8a6 (0xb3e6 bytes) 000248 1713296863.937430 write 0x00e1b3 thru 0x01694b (0x8799 bytes) 000249 1713296863.939934 trunc from 0x03e8a7 to 0x032e4a (0xba5c bytes) 000250 1713296863.943468 write 0x03702f thru 0x03ffff (0x8fd1 bytes) 000252 1713296863.944413 fallocate 0x0040fe thru 0x006507 (0x240a bytes) 000253 1713296863.945820 write 0x02986a thru 0x0308a2 (0x7039 bytes) 000254 1713296863.946584 punch from 0x00011a to 0x000e8b (0xd72 bytes) 000255 1713296863.947740 punch from 0x0121db to 0x012dd2 (0xbf8 bytes) 000256 1713296863.951204 write 0x0142f8 thru 0x015e24 (0x1b2d bytes) 000257 1713296863.952918 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000258 1713296863.995354 fallocate 0x024989 thru 0x034704 (0xfd7c bytes) 000259 1713296863.996779 mapwrite 0x02eb44 thru 0x030dbc (0x2279 bytes) 000260 1713296864.005295 mapread 0x01f87f thru 0x02b1fa (0xb97c bytes) 000261 1713296864.008872 punch from 0x010f60 to 0x01f214 (0xe2b5 bytes) 000262 1713296864.011721 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000265 1713296864.050248 punch from 0x03cf07 to 0x03ffff (0x30f9 bytes) 000266 1713296864.051811 mapwrite 0x02dd56 thru 0x038e0a (0xb0b5 bytes) 000268 1713296864.063082 punch from 0x038ecb to 0x03e9b3 (0x5ae9 bytes) 000269 1713296864.064559 mapread 0x035dcf thru 0x03ffff (0xa231 bytes) 000270 1713296864.066798 fallocate 0x02d3c4 thru 0x02d45f (0x9c bytes) 000271 1713296864.069094 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000273 1713296864.114851 fallocate 0x0078f7 thru 0x0102e3 (0x89ed bytes) 000275 1713296864.117112 read 0x0324eb thru 0x033099 (0xbaf bytes) 000276 1713296864.118679 mapwrite 0x003c90 thru 0x0058bf (0x1c30 bytes) 000277 1713296864.126053 mapread 0x01eba0 thru 0x026056 (0x74b7 bytes) 000278 1713296864.129774 fallocate 0x004f33 thru 0x009856 (0x4924 bytes) 000280 1713296864.131596 write 0x000741 thru 0x00dd54 (0xd614 bytes) 000281 1713296864.134882 write 0x004ffe thru 0x00801a (0x301d bytes) 000282 1713296864.135342 write 0x039259 thru 0x03ffff (0x6da7 bytes) 000283 1713296864.137631 punch from 0x03d8f8 to 0x03ffff (0x2708 bytes) 000285 1713296864.141804 write 0x01873e thru 0x01e3b5 (0x5c78 bytes) 000286 1713296864.143237 trunc from 0x040000 to 0x0097be (0x36841 bytes) 000287 1713296864.147846 read 0x0024a3 thru 0x0097be (0x731c bytes) 000288 1713296864.148160 read 0x004e46 thru 0x0097be (0x4979 bytes) 000289 1713296864.148480 write 0x023c53 thru 0x025057 (0x1405 bytes) 000290 1713296864.148964 read 0x00f9a7 thru 0x01e794 (0xedee bytes) 000291 1713296864.152492 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000293 1713296864.181826 read 0x01a8c1 thru 0x0248a6 (0x9fe6 bytes) 000295 1713296864.184353 read 0x00efbb thru 0x01e245 (0xf28b bytes) 000299 1713296864.187615 write 0x038650 thru 0x03c3ce (0x3d7f bytes) 000301 1713296864.189583 fallocate 0x0347db thru 0x03fb17 (0xb33d bytes) 000302 1713296864.191110 trunc from 0x03fb18 to 0x01d5e2 (0x22535 bytes) 000303 1713296864.194676 trunc from 0x01d5e3 to 0x02628b (0x8ca9 bytes) 000304 1713296864.199068 MIRROR_EXTEND to 2 mirrors 000305 1713296864.293256 mapread 0x01d41b thru 0x023659 (0x623f bytes) 000306 1713296864.297302 read 0x012152 thru 0x014b43 (0x29f2 bytes) 000309 1713296864.298867 fallocate 0x03235d thru 0x037a2b (0x56cf bytes) 000310 1713296864.307205 write 0x03dbd6 thru 0x03ffff (0x242a bytes) 000311 1713296864.308802 mapwrite 0x019a9d thru 0x02448d (0xa9f1 bytes) 000312 1713296864.319014 write 0x027b80 thru 0x0352af (0xd730 bytes) 000313 1713296864.322070 MIRROR_RESYNC 2 mirrors 000315 1713296864.422219 fallocate 0x015b7b thru 0x022cc5 (0xd14b bytes) 000316 1713296864.429602 mapread 0x01a0f3 thru 0x027d9e (0xdcac bytes) 000317 1713296864.435673 fallocate 0x018c54 thru 0x01fe34 (0x71e1 bytes) 000318 1713296864.437395 mapread 0x009361 thru 0x0144db (0xb17b bytes) 000319 1713296864.441364 fallocate 0x013f27 thru 0x015ed0 (0x1faa bytes) 000320 1713296864.442812 mapread 0x03e79c thru 0x03ffff (0x1864 bytes) 000321 1713296864.444913 mapread 0x00883d thru 0x00d11e (0x48e2 bytes) 000322 1713296864.445485 fallocate 0x0207ce thru 0x0245c7 (0x3dfa bytes) 000323 1713296864.446833 mapread 0x02bcbb thru 0x038f4b (0xd291 bytes) 000324 1713296864.450335 write 0x024ec8 thru 0x0301cc (0xb305 bytes) 000325 1713296864.453564 MIRROR_RESYNC 2 mirrors 000326 1713296864.545556 mapwrite 0x02fd80 thru 0x03c62f (0xc8b0 bytes) 000327 1713296864.575563 read 0x024c41 thru 0x02d879 (0x8c39 bytes) 000328 1713296864.577482 mapwrite 0x038fa0 thru 0x03dbea (0x4c4b bytes) 000329 1713296864.581790 fallocate 0x0001d7 thru 0x0061f0 (0x601a bytes) 000330 1713296864.583294 punch from 0x017a23 to 0x01acad (0x328b bytes) 000331 1713296864.584634 write 0x030d42 thru 0x036eee (0x61ad bytes) 000332 1713296864.586364 trunc from 0x040000 to 0x02cb16 (0x134e9 bytes) 000333 1713296864.596870 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000334 1713296864.624753 write 0x01174f thru 0x01a6a7 (0x8f59 bytes) 000335 1713296864.628766 fallocate 0x02b9b6 thru 0x03b414 (0xfa5f bytes) 000336 1713296864.630553 write 0x030cf4 thru 0x034b41 (0x3e4e bytes) 000337 1713296864.633330 read 0x015de9 thru 0x023c4d (0xde65 bytes) 000338 1713296864.635399 write 0x01df07 thru 0x02624c (0x8346 bytes) 000339 1713296864.637096 mapread 0x0320e1 thru 0x03b414 (0x9334 bytes) 000340 1713296864.638805 fallocate 0x013189 thru 0x01c618 (0x9490 bytes) 000341 1713296864.640237 punch from 0x00bcbc to 0x00fbea (0x3f2f bytes) 000342 1713296864.641614 write 0x027352 thru 0x02d337 (0x5fe6 bytes) 000343 1713296864.643246 mapwrite 0x01f769 thru 0x0273bc (0x7c54 bytes) 000345 1713296864.650058 write 0x022b94 thru 0x028ec0 (0x632d bytes) 000346 1713296864.650785 fallocate 0x03e793 thru 0x03ffff (0x186d bytes) 000348 1713296864.652418 trunc from 0x03b415 to 0x02a1f8 (0x1121c bytes) 000349 1713296864.656235 read 0x01575d thru 0x02387d (0xe121 bytes) 000350 1713296864.656585 mapwrite 0x02a5e4 thru 0x0356b6 (0xb0d3 bytes) 000351 1713296864.668415 write 0x0192ce thru 0x02661d (0xd350 bytes) 000353 1713296864.669391 write 0x03121a thru 0x03d6bf (0xc4a6 bytes) 000354 1713296864.670088 trunc from 0x03d6c0 to 0x0390bd (0x4602 bytes) 000355 1713296864.673578 trunc from 0x0390be to 0x01b07d (0x1e040 bytes) 000356 1713296864.677007 punch from 0x0027ba to 0x008fc5 (0x680c bytes) 000358 1713296864.678941 trunc from 0x01b07e to 0x038752 (0x1d6d5 bytes) 000359 1713296864.682131 read 0x0256aa thru 0x02ce2c (0x7783 bytes) 000360 1713296864.684746 write 0x009f99 thru 0x019ab0 (0xfb18 bytes) 000361 1713296864.687205 MIRROR_EXTEND to 2 mirrors 000362 1713296864.794888 mapwrite 0x0322b1 thru 0x03ffff (0xdd4f bytes) 000364 1713296864.819882 punch from 0x00d95e to 0x014cf3 (0x7396 bytes) 000366 1713296864.821980 fallocate 0x03dc93 thru 0x03ffff (0x236d bytes) 000367 1713296864.823500 trunc from 0x040000 to 0x027a85 (0x1857a bytes) 000368 1713296864.828488 punch from 0x021371 to 0x02581a (0x44aa bytes) 000369 1713296864.830002 read 0x00426e thru 0x00d84c (0x95df bytes) 000370 1713296864.832498 mapread 0x002f63 thru 0x012d56 (0xfdf4 bytes) 000371 1713296864.837631 MIRROR_EXTEND to 3 mirrors 000372 1713296864.949437 write 0x03ccb2 thru 0x03ffff (0x334e bytes) 000373 1713296864.958014 trunc from 0x040000 to 0x0066f1 (0x3990e bytes) 000374 1713296864.966739 fallocate 0x017a51 thru 0x027955 (0xff05 bytes) 000376 1713296864.968461 mapread 0x003c6c thru 0x009e70 (0x6205 bytes) 000377 1713296864.972932 trunc from 0x027956 to 0x02aef7 (0x35a2 bytes) 000379 1713296864.976902 trunc from 0x02aef8 to 0x001447 (0x29ab0 bytes) 000380 1713296864.980327 mapread 0x001420 thru 0x001447 (0x28 bytes) 000381 1713296864.980903 trunc from 0x001448 to 0x03bb20 (0x3a6d9 bytes) 000385 1713296864.984720 write 0x03530e thru 0x03ffff (0xacf2 bytes) 000387 1713296864.987603 mapwrite 0x03663c thru 0x03ffff (0x99c4 bytes) 000388 1713296864.992958 mapread 0x03aba0 thru 0x03d227 (0x2688 bytes) 000389 1713296864.993514 read 0x0379e1 thru 0x03ffff (0x861f bytes) 000390 1713296864.993846 fallocate 0x030655 thru 0x038dc1 (0x876d bytes) 000391 1713296864.995373 mapwrite 0x0103d7 thru 0x017a29 (0x7653 bytes) 000392 1713296865.002207 fallocate 0x017ba3 thru 0x0195e4 (0x1a42 bytes) 000393 1713296865.003464 punch from 0x00e1db to 0x017a52 (0x9878 bytes) 000395 1713296865.005294 fallocate 0x0330b0 thru 0x03ffff (0xcf50 bytes) 000396 1713296865.006797 write 0x027726 thru 0x02d3e8 (0x5cc3 bytes) 000398 1713296865.009593 punch from 0x026f70 to 0x032368 (0xb3f9 bytes) 000399 1713296865.012816 punch from 0x003766 to 0x00b893 (0x812e bytes) 000400 1713296865.014003 mapread 0x00b987 thru 0x016fa6 (0xb620 bytes) 000401 1713296865.018406 write 0x018889 thru 0x0264d6 (0xdc4e bytes) 000402 1713296865.020323 fallocate 0x02a7cf thru 0x02e8ad (0x40df bytes) 000403 1713296865.021728 mapread 0x02ec63 thru 0x033936 (0x4cd4 bytes) 000404 1713296865.023810 mapwrite 0x012f00 thru 0x0181d8 (0x52d9 bytes) 000406 1713296865.029168 mapwrite 0x032a5c thru 0x03f678 (0xcc1d bytes) 000407 1713296865.036330 MIRROR_RESYNC 3 mirrors 000408 1713296865.148153 punch from 0x0060df to 0x0133e3 (0xd305 bytes) 000409 1713296865.156267 read 0x02da4e thru 0x036cf3 (0x92a6 bytes) 000410 1713296865.158806 mapread 0x00772c thru 0x011f6f (0xa844 bytes) 000411 1713296865.164960 mapread 0x02645f thru 0x02af9b (0x4b3d bytes) 000412 1713296865.167193 write 0x026806 thru 0x02c158 (0x5953 bytes) 000413 1713296865.168975 mapwrite 0x02aecf thru 0x032c7e (0x7db0 bytes) 000414 1713296865.176180 MIRROR_RESYNC 3 mirrors 000415 1713296865.267539 fallocate 0x030e9b thru 0x03a222 (0x9388 bytes) 000417 1713296865.276130 write 0x025d3b thru 0x026b3a (0xe00 bytes) 000418 1713296865.279877 read 0x00df5b thru 0x016643 (0x86e9 bytes) 000419 1713296865.281891 fallocate 0x01ff86 thru 0x022bfe (0x2c79 bytes) 000420 1713296865.283356 mapread 0x00ac05 thru 0x0181b5 (0xd5b1 bytes) 000421 1713296865.287507 read 0x001710 thru 0x00aaf0 (0x93e1 bytes) 000422 1713296865.290106 fallocate 0x0134ef thru 0x01b3cd (0x7edf bytes) 000423 1713296865.291770 fallocate 0x033645 thru 0x036759 (0x3115 bytes) 000424 1713296865.294419 MIRROR_EXTEND to 4 mirrors 000426 1713296865.404142 punch from 0x007c9c to 0x016c94 (0xeff9 bytes) 000427 1713296865.412168 write 0x0298e2 thru 0x02bbe1 (0x2300 bytes) 000429 1713296865.416128 punch from 0x014ab8 to 0x0216fe (0xcc47 bytes) 000430 1713296865.417814 mapwrite 0x000f90 thru 0x001af0 (0xb61 bytes) 000432 1713296865.425033 read 0x018049 thru 0x0261b8 (0xe170 bytes) 000433 1713296865.427625 trunc from 0x040000 to 0x00569d (0x3a962 bytes) 000434 1713296865.436215 write 0x026243 thru 0x034f13 (0xecd1 bytes) 000436 1713296865.437410 punch from 0x00b873 to 0x010d22 (0x54b0 bytes) 000440 1713296865.439920 write 0x03fcaf thru 0x03ffff (0x351 bytes) 000441 1713296865.440600 trunc from 0x040000 to 0x03c8c9 (0x3736 bytes) 000442 1713296865.444552 mapread 0x020a11 thru 0x02f83a (0xee2a bytes) 000443 1713296865.446858 punch from 0x0268ed to 0x0366ea (0xfdfe bytes) 000444 1713296865.450598 punch from 0x02f1b6 to 0x031661 (0x24ac bytes) 000445 1713296865.451798 read 0x00708a thru 0x007494 (0x40b bytes) 000447 1713296865.454886 write 0x02e23e thru 0x032e24 (0x4be7 bytes) 000449 1713296865.455782 punch from 0x02e756 to 0x03c8c9 (0xe174 bytes) 000451 1713296865.459244 punch from 0x001e05 to 0x003887 (0x1a83 bytes) 000452 1713296865.460800 fallocate 0x0207a9 thru 0x023733 (0x2f8b bytes) 000454 1713296865.462618 fallocate 0x026069 thru 0x02e484 (0x841c bytes) 000455 1713296865.464260 read 0x03b107 thru 0x03c8c9 (0x17c3 bytes) 000456 1713296865.467114 punch from 0x00fa21 to 0x013b04 (0x40e4 bytes) 000457 1713296865.468609 mapwrite 0x03e235 thru 0x03ffff (0x1dcb bytes) 000459 1713296865.478226 read 0x010ede thru 0x01bef9 (0xb01c bytes) 000461 1713296865.480540 mapread 0x013b81 thru 0x019e6c (0x62ec bytes) 000462 1713296865.480908 mapwrite 0x035559 thru 0x03e136 (0x8bde bytes) 000463 1713296865.485356 trunc from 0x040000 to 0x00fdb9 (0x30246 bytes) 000465 1713296865.489246 punch from 0x003b58 to 0x00fdb9 (0xc262 bytes) 000466 1713296865.490597 mapwrite 0x0388db thru 0x03ffff (0x7725 bytes) 000467 1713296865.507666 write 0x01ad11 thru 0x02a077 (0xf367 bytes) 000468 1713296865.511860 MIRROR_RESYNC 4 mirrors 000469 1713296865.656535 trunc from 0x040000 to 0x03edae (0x1251 bytes) 000471 1713296865.672567 write 0x023979 thru 0x031923 (0xdfab bytes) 000473 1713296865.677564 MIRROR_EXTEND to 5 mirrors 000474 1713296865.786573 write 0x028098 thru 0x030c2f (0x8b98 bytes) 000475 1713296865.797741 mapread 0x00be34 thru 0x012cd1 (0x6e9e bytes) 000476 1713296865.800373 mapwrite 0x038bfa thru 0x03cb44 (0x3f4b bytes) 000477 1713296865.808561 trunc from 0x03edaf to 0x002262 (0x3cb4c bytes) 000480 1713296865.817611 read 0x000ce5 thru 0x002262 (0x157e bytes) 000481 1713296865.819292 write 0x0344ec thru 0x03ad77 (0x688c bytes) 000482 1713296865.819947 mapwrite 0x0272a1 thru 0x02a410 (0x3170 bytes) 000484 1713296865.827315 write 0x01b381 thru 0x01cc0d (0x188d bytes) 000485 1713296865.829654 write 0x01030c thru 0x010798 (0x48d bytes) 000487 1713296865.832163 mapread 0x02822f thru 0x02ea6f (0x6841 bytes) 000488 1713296865.832572 mapread 0x006b0b thru 0x012643 (0xbb39 bytes) 000490 1713296865.836352 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000491 1713296865.882734 read 0x006cde thru 0x00deae (0x71d1 bytes) 000493 1713296865.885418 read 0x02c2c6 thru 0x0353fb (0x9136 bytes) 000495 1713296865.888676 read 0x019fd2 thru 0x01a935 (0x964 bytes) 000499 1713296865.889843 punch from 0x02aaf6 to 0x02c09e (0x15a9 bytes) 000500 1713296865.892569 MIRROR_EXTEND to 5 mirrors 000501 1713296866.005466 read 0x013493 thru 0x021637 (0xe1a5 bytes) 000502 1713296866.009398 punch from 0x02d7c3 to 0x034dad (0x75eb bytes) 000503 1713296866.018159 punch from 0x003098 to 0x00e46e (0xb3d7 bytes) 000504 1713296866.019441 mapwrite 0x0153c4 thru 0x01c43f (0x707c bytes) 000505 1713296866.030334 MIRROR_RESYNC 5 mirrors 000506 1713296866.270103 MIRROR_EXTEND to 6 mirrors 000507 1713296866.383895 mapwrite 0x03a0db thru 0x03ffff (0x5f25 bytes) 000508 1713296866.407925 trunc from 0x040000 to 0x038f12 (0x70ed bytes) 000509 1713296866.412226 mapread 0x004d80 thru 0x0098f2 (0x4b73 bytes) 000510 1713296866.414775 mapread 0x01596c thru 0x01bf5a (0x65ef bytes) 000511 1713296866.417278 mapread 0x01a772 thru 0x0273cd (0xcc5c bytes) 000512 1713296866.419540 fallocate 0x03aab1 thru 0x03ffff (0x554f bytes) 000513 1713296866.420949 mapread 0x034ece thru 0x038f12 (0x4045 bytes) 000514 1713296866.423220 mapread 0x025be0 thru 0x02be99 (0x62ba bytes) 000515 1713296866.423659 read 0x0212a4 thru 0x02e2e3 (0xd040 bytes) 000516 1713296866.425467 MIRROR_RESYNC 6 mirrors 000517 1713296866.704969 punch from 0x03175c to 0x032472 (0xd17 bytes) 000519 1713296866.714125 read 0x00533b thru 0x00d890 (0x8556 bytes) 000521 1713296866.716837 trunc from 0x038f13 to 0x00bb8a (0x2d388 bytes) 000522 1713296866.725639 mapwrite 0x01506d thru 0x023d87 (0xed1b bytes) 000523 1713296866.742388 MIRROR_EXTEND to 7 mirrors 000524 1713296866.840275 mapwrite 0x03af67 thru 0x03ffff (0x5099 bytes) 000525 1713296866.864150 read 0x022fc5 thru 0x031a8f (0xeacb bytes) 000526 1713296866.866643 trunc from 0x040000 to 0x024d57 (0x1b2a8 bytes) 000527 1713296866.872037 mapwrite 0x02373f thru 0x023a25 (0x2e7 bytes) 000528 1713296866.877206 read 0x023690 thru 0x024d57 (0x16c8 bytes) 000529 1713296866.878884 MIRROR_RESYNC 7 mirrors 000530 1713296867.095591 punch from 0x0129b4 to 0x014c7a (0x22c7 bytes) 000531 1713296867.102690 write 0x003526 thru 0x00c90b (0x93e6 bytes) 000532 1713296867.105645 read 0x0032fa thru 0x00437d (0x1084 bytes) 000533 1713296867.106121 mapread 0x00151a thru 0x00fe8d (0xe974 bytes) 000534 1713296867.109799 punch from 0x021b34 to 0x024d57 (0x3224 bytes) 000535 1713296867.111365 fallocate 0x00594c thru 0x0061d3 (0x888 bytes) 000536 1713296867.112682 mapread 0x00124e thru 0x010772 (0xf525 bytes) 000537 1713296867.113145 mapwrite 0x031010 thru 0x035898 (0x4889 bytes) 000539 1713296867.127379 MIRROR_RESYNC 7 mirrors 000540 1713296867.410920 mapwrite 0x026767 thru 0x029dd7 (0x3671 bytes) 000541 1713296867.441309 MIRROR_EXTEND to 8 mirrors 000542 1713296867.563397 read 0x031831 thru 0x035898 (0x4068 bytes) 000543 1713296867.565067 read 0x018cd4 thru 0x019249 (0x576 bytes) 000544 1713296867.567913 trunc from 0x035899 to 0x03adc6 (0x552e bytes) 000545 1713296867.583262 mapread 0x014c7d thru 0x018f9f (0x4323 bytes) 000547 1713296867.590862 MIRROR_RESYNC 8 mirrors 000548 1713296868.000407 read 0x037aae thru 0x03adc6 (0x3319 bytes) 000549 1713296868.002906 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000550 1713296868.072343 read 0x029149 thru 0x030e05 (0x7cbd bytes) 000551 1713296868.074555 write 0x025afb thru 0x02df6f (0x8475 bytes) 000552 1713296868.085328 read 0x02bad0 thru 0x031663 (0x5b94 bytes) 000553 1713296868.088287 punch from 0x012263 to 0x0197b7 (0x7555 bytes) 000554 1713296868.090156 fallocate 0x026b06 thru 0x031d26 (0xb221 bytes) 000555 1713296868.091796 mapwrite 0x002943 thru 0x008c80 (0x633e bytes) 000556 1713296868.098202 punch from 0x0344fc to 0x03adc6 (0x68cb bytes) 000557 1713296868.099414 mapwrite 0x033362 thru 0x03488d (0x152c bytes) 000558 1713296868.104199 fallocate 0x003108 thru 0x012887 (0xf780 bytes) 000559 1713296868.105813 trunc from 0x03adc7 to 0x02b282 (0xfb44 bytes) 000560 1713296868.114745 read 0x002bad thru 0x0115ee (0xea42 bytes) 000561 1713296868.116432 MIRROR_EXTEND to 8 mirrors 000562 1713296868.221923 read 0x00e266 thru 0x01e11b (0xfeb6 bytes) 000563 1713296868.224304 trunc from 0x02b283 to 0x002c16 (0x2866c bytes) 000564 1713296868.239686 MIRROR_EXTEND to 9 mirrors 000565 1713296868.324040 trunc from 0x002c17 to 0x034e65 (0x3224f bytes) 000566 1713296868.341473 write 0x02e435 thru 0x03bea9 (0xda75 bytes) 000567 1713296868.343657 mapwrite 0x03d603 thru 0x03ffff (0x29fd bytes) 000568 1713296868.356620 mapwrite 0x019b93 thru 0x024ece (0xb33c bytes) 000569 1713296868.366751 mapwrite 0x01366b thru 0x02260e (0xefa4 bytes) 000570 1713296868.373474 trunc from 0x040000 to 0x00e854 (0x317ab bytes) 000571 1713296868.378826 MIRROR_RESYNC 9 mirrors 000572 1713296868.568641 MIRROR_EXTEND to 10 mirrors 000573 1713296868.684723 write 0x01bcf1 thru 0x0217a1 (0x5ab1 bytes) 000574 1713296868.693862 punch from 0x0163ef to 0x0207e7 (0xa3f9 bytes) 000575 1713296868.697302 read 0x00403a thru 0x006a47 (0x2a0e bytes) 000576 1713296868.699840 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000577 1713296868.744755 mapwrite 0x03fc9f thru 0x03ffff (0x361 bytes) 000579 1713296868.758422 fallocate 0x026dfc thru 0x03124d (0xa452 bytes) 000580 1713296868.760217 read 0x00c6b8 thru 0x01a9b0 (0xe2f9 bytes) 000581 1713296868.762940 punch from 0x00ccec to 0x018ac2 (0xbdd7 bytes) 000584 1713296868.765127 write 0x031e26 thru 0x03754f (0x572a bytes) 000585 1713296868.768396 mapwrite 0x02a99a thru 0x0308d9 (0x5f40 bytes) 000588 1713296868.777198 MIRROR_EXTEND to 10 mirrors 000589 1713296868.894654 mapread 0x039677 thru 0x03e5aa (0x4f34 bytes) 000590 1713296868.902421 fallocate 0x0084ba thru 0x00fc36 (0x777d bytes) 000591 1713296868.913215 MIRROR_RESYNC 10 mirrors 000592 1713296869.255187 fallocate 0x01f10e thru 0x022f2f (0x3e22 bytes) 000593 1713296869.265252 MIRROR_EXTEND to 11 mirrors 000594 1713296869.389345 write 0x012cea thru 0x01f2ae (0xc5c5 bytes) 000596 1713296869.402334 punch from 0x030790 to 0x0354cc (0x4d3d bytes) 000597 1713296869.405196 MIRROR_EXTEND to 12 mirrors 000598 1713296869.541071 mapwrite 0x02aca8 thru 0x038572 (0xd8cb bytes) 000599 1713296869.576813 mapread 0x03ee62 thru 0x03ffff (0x119e bytes) 000600 1713296869.578936 write 0x032c2d thru 0x039c49 (0x701d bytes) 000602 1713296869.582138 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000603 1713296869.630425 mapread 0x0043a5 thru 0x00fd2a (0xb986 bytes) 000604 1713296869.636699 mapread 0x009144 thru 0x015333 (0xc1f0 bytes) 000605 1713296869.637170 write 0x02d351 thru 0x030bcd (0x387d bytes) 000606 1713296869.640564 mapread 0x02f379 thru 0x03d240 (0xdec8 bytes) 000607 1713296869.643048 fallocate 0x0391f0 thru 0x03ffff (0x6e10 bytes) 000608 1713296869.644709 read 0x032019 thru 0x03ba25 (0x9a0d bytes) 000609 1713296869.645065 fallocate 0x003845 thru 0x006b80 (0x333c bytes) 000610 1713296869.646315 mapwrite 0x013cd9 thru 0x014aa1 (0xdc9 bytes) 000611 1713296869.650710 mapwrite 0x01b591 thru 0x01d4dd (0x1f4d bytes) 000612 1713296869.657161 mapread 0x007703 thru 0x010c73 (0x9571 bytes) 000613 1713296869.658856 MIRROR_EXTEND to 12 mirrors 000614 1713296869.795157 read 0x00a644 thru 0x011e85 (0x7842 bytes) 000616 1713296869.800978 mapread 0x030fea thru 0x039d19 (0x8d30 bytes) 000617 1713296869.808698 write 0x008d7c thru 0x015fc1 (0xd246 bytes) 000618 1713296869.820644 trunc from 0x040000 to 0x026aa4 (0x1955b bytes) 000619 1713296869.825008 punch from 0x0125bb to 0x01d332 (0xad78 bytes) 000622 1713296869.828901 fallocate 0x0332ff thru 0x03cd9e (0x9aa0 bytes) 000623 1713296869.830358 punch from 0x029999 to 0x035d83 (0xc3eb bytes) 000626 1713296869.832517 mapread 0x019e7f thru 0x019f0f (0x91 bytes) 000627 1713296869.835532 read 0x0192e8 thru 0x01f6bb (0x63d4 bytes) 000628 1713296869.836006 write 0x028575 thru 0x030920 (0x83ac bytes) 000629 1713296869.839100 mapwrite 0x039c47 thru 0x03c47b (0x2835 bytes) 000630 1713296869.845298 mapwrite 0x031a32 thru 0x033e3a (0x2409 bytes) 000631 1713296869.851160 read 0x00837f thru 0x00e769 (0x63eb bytes) 000632 1713296869.851560 mapread 0x0166e5 thru 0x02070c (0xa028 bytes) 000633 1713296869.852063 punch from 0x00dca0 to 0x01c5eb (0xe94c bytes) 000634 1713296869.854532 punch from 0x02629e to 0x028c62 (0x29c5 bytes) 000636 1713296869.856460 punch from 0x028132 to 0x0334dd (0xb3ac bytes) 000637 1713296869.858056 read 0x0244fc thru 0x02b8ab (0x73b0 bytes) 000638 1713296869.861023 read 0x019f84 thru 0x026dbd (0xce3a bytes) 000641 1713296869.861801 mapwrite 0x01ccce thru 0x024d90 (0x80c3 bytes) 000642 1713296869.868374 read 0x033d96 thru 0x036b94 (0x2dff bytes) 000643 1713296869.868848 mapread 0x03910e thru 0x03b58c (0x247f bytes) 000644 1713296869.869505 read 0x01a822 thru 0x023b0b (0x92ea bytes) 000645 1713296869.869985 trunc from 0x03cd9f to 0x001251 (0x3bb4d bytes) 000646 1713296869.874276 punch from 0x000425 to 0x001251 (0xe2d bytes) 000647 1713296869.875813 fallocate 0x03b960 thru 0x03ffff (0x46a0 bytes) 000648 1713296869.878639 mapwrite 0x0048f4 thru 0x00fffa (0xb707 bytes) 000649 1713296869.889354 trunc from 0x040000 to 0x03a806 (0x57f9 bytes) 000653 1713296869.895096 MIRROR_RESYNC 12 mirrors 000654 1713296870.256262 fallocate 0x025476 thru 0x029ca8 (0x4833 bytes) 000655 1713296870.264569 mapread 0x0037a4 thru 0x0053ba (0x1c17 bytes) 000656 1713296870.268858 mapwrite 0x00c47a thru 0x00d32d (0xeb4 bytes) 000657 1713296870.273618 fallocate 0x020636 thru 0x023f91 (0x395c bytes) 000658 1713296870.274882 read 0x02c87b thru 0x037ab5 (0xb23b bytes) 000659 1713296870.277158 mapread 0x019aa4 thru 0x026300 (0xc85d bytes) 000660 1713296870.281162 fallocate 0x035a2f thru 0x03ee22 (0x93f4 bytes) 000661 1713296870.282602 read 0x032bd2 thru 0x032e9c (0x2cb bytes) 000662 1713296870.282900 mapwrite 0x017445 thru 0x019e11 (0x29cd bytes) 000663 1713296870.286320 mapwrite 0x023046 thru 0x0254bc (0x2477 bytes) 000664 1713296870.289761 mapread 0x0344e9 thru 0x03ba33 (0x754b bytes) 000665 1713296870.293269 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000667 1713296870.341375 read 0x0032dc thru 0x0118fc (0xe621 bytes) 000668 1713296870.343965 fallocate 0x0182c2 thru 0x024a3f (0xc77e bytes) 000669 1713296870.345459 read 0x038e71 thru 0x03be1c (0x2fac bytes) 000670 1713296870.348688 write 0x02cbae thru 0x035a5a (0x8ead bytes) 000672 1713296870.351063 read 0x0254cf thru 0x0272ae (0x1de0 bytes) 000673 1713296870.351326 write 0x038fe1 thru 0x039884 (0x8a4 bytes) 000674 1713296870.351721 mapwrite 0x03aed3 thru 0x03ffff (0x512d bytes) 000675 1713296870.362761 fallocate 0x03260a thru 0x0391de (0x6bd5 bytes) 000679 1713296870.364925 read 0x02fd3c thru 0x02ff14 (0x1d9 bytes) 000680 1713296870.365325 fallocate 0x032817 thru 0x03ffff (0xd7e9 bytes) 000681 1713296870.366679 mapwrite 0x03113c thru 0x03229d (0x1162 bytes) 000683 1713296870.370197 write 0x022281 thru 0x0288ef (0x666f bytes) 000684 1713296870.370901 punch from 0x03ec36 to 0x03ffff (0x13ca bytes) 000685 1713296870.372372 mapwrite 0x02b211 thru 0x038e41 (0xdc31 bytes) 000686 1713296870.379004 fallocate 0x03d776 thru 0x03df96 (0x821 bytes) 000687 1713296870.380863 mapread 0x0023b0 thru 0x00264e (0x29f bytes) 000688 1713296870.381344 mapread 0x03ece8 thru 0x03ffff (0x1318 bytes) 000689 1713296870.383249 trunc from 0x040000 to 0x0046c0 (0x3b93f bytes) 000690 1713296870.387857 read 0x001e14 thru 0x0046c0 (0x28ad bytes) 000691 1713296870.388255 punch from 0x003286 to 0x0046c0 (0x143b bytes) 000692 1713296870.389805 mapread 0x001308 thru 0x0046c0 (0x33b9 bytes) 000693 1713296870.391727 fallocate 0x00cb10 thru 0x00e16c (0x165d bytes) 000696 1713296870.394095 punch from 0x003005 to 0x0046c0 (0x16bc bytes) 000697 1713296870.396974 MIRROR_EXTEND to 12 mirrors 000698 1713296870.498950 trunc from 0x0046c1 to 0x001ffd (0x26c3 bytes) 000699 1713296870.514707 fallocate 0x008353 thru 0x0128f6 (0xa5a4 bytes) 000700 1713296870.516281 punch from 0x001725 to 0x001ffd (0x8d9 bytes) 000701 1713296870.517572 mapwrite 0x015624 thru 0x01c824 (0x7201 bytes) 000702 1713296870.533021 mapwrite 0x01007c thru 0x019e4a (0x9dcf bytes) 000703 1713296870.538617 punch from 0x00b952 to 0x01311b (0x77ca bytes) 000704 1713296870.540247 write 0x025570 thru 0x02ac59 (0x56ea bytes) 000705 1713296870.541008 punch from 0x0114c4 to 0x016255 (0x4d92 bytes) 000707 1713296870.543005 trunc from 0x02ac5a to 0x023673 (0x75e6 bytes) 000708 1713296870.547547 mapread 0x01446f thru 0x01af22 (0x6ab4 bytes) 000709 1713296870.551288 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000710 1713296870.598366 punch from 0x00bb43 to 0x01617f (0xa63d bytes) 000712 1713296870.600338 trunc from 0x023674 to 0x03fb3e (0x1c4cb bytes) 000713 1713296870.605295 mapread 0x01b7c3 thru 0x0207d7 (0x5015 bytes) 000714 1713296870.609836 trunc from 0x03fb3f to 0x00063f (0x3f4ff bytes) 000715 1713296870.613546 mapread 0x00020a thru 0x00063f (0x436 bytes) 000716 1713296870.614938 read 0x00010a thru 0x00063f (0x536 bytes) 000717 1713296870.616207 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000718 1713296870.658920 punch from 0x00007a to 0x00063f (0x5c6 bytes) 000719 1713296870.660533 fallocate 0x007390 thru 0x008b11 (0x1782 bytes) 000720 1713296870.661892 read 0x0066fe thru 0x008b11 (0x2414 bytes) 000721 1713296870.663634 mapread 0x0004c5 thru 0x008b11 (0x864d bytes) 000722 1713296870.668567 fallocate 0x024cb4 thru 0x02a683 (0x59d0 bytes) 000723 1713296870.670333 mapwrite 0x02a118 thru 0x0351a7 (0xb090 bytes) 000724 1713296870.685196 write 0x025ee5 thru 0x034bbe (0xecda bytes) 000725 1713296870.685973 write 0x02a0b5 thru 0x0382e5 (0xe231 bytes) 000726 1713296870.687047 write 0x0093d1 thru 0x01649b (0xd0cb bytes) 000727 1713296870.690917 write 0x006db2 thru 0x01476a (0xd9b9 bytes) 000728 1713296870.691660 mapread 0x024be7 thru 0x033adf (0xeef9 bytes) 000729 1713296870.692134 write 0x02e6d8 thru 0x03c66a (0xdf93 bytes) 000730 1713296870.692980 mapwrite 0x02058c thru 0x02702a (0x6a9f bytes) 000731 1713296870.701645 write 0x026999 thru 0x027f74 (0x15dc bytes) 000732 1713296870.702110 write 0x0084fb thru 0x017aff (0xf605 bytes) 000733 1713296870.704205 punch from 0x00bdc4 to 0x01a7f3 (0xea30 bytes) 000734 1713296870.708979 mapwrite 0x027275 thru 0x036ee7 (0xfc73 bytes) 000735 1713296870.714864 mapread 0x012e1d thru 0x017a8d (0x4c71 bytes) 000737 1713296870.717853 trunc from 0x03c66b to 0x0131be (0x294ac bytes) 000738 1713296870.722324 fallocate 0x01d6bf thru 0x025412 (0x7d54 bytes) 000739 1713296870.723829 trunc from 0x0131bf to 0x03b95f (0x287a1 bytes) 000740 1713296870.727478 punch from 0x02a03c to 0x036c5d (0xcc22 bytes) 000741 1713296870.728831 punch from 0x027a81 to 0x028edb (0x145b bytes) 000742 1713296870.730214 read 0x03064e thru 0x03b95f (0xb312 bytes) 000744 1713296870.734007 mapwrite 0x00e642 thru 0x01698c (0x834b bytes) 000745 1713296870.741375 trunc from 0x03b960 to 0x032fb8 (0x89a7 bytes) 000747 1713296870.746718 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000748 1713296870.797341 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000749 1713296870.844647 punch from 0x028d8b to 0x031f63 (0x91d9 bytes) 000750 1713296870.846227 read 0x01a052 thru 0x0221d8 (0x8187 bytes) 000751 1713296870.848532 read 0x02ce68 thru 0x032fb8 (0x6151 bytes) 000752 1713296870.851373 mapwrite 0x030846 thru 0x03780f (0x6fca bytes) 000753 1713296870.863934 fallocate 0x0164b1 thru 0x01887d (0x23cd bytes) 000755 1713296870.865812 trunc from 0x037810 to 0x038731 (0xf22 bytes) 000756 1713296870.869433 mapwrite 0x0196f7 thru 0x019857 (0x161 bytes) 000757 1713296870.873552 punch from 0x002e80 to 0x00f3a0 (0xc521 bytes) 000758 1713296870.876046 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000760 1713296870.922626 mapwrite 0x0310f8 thru 0x035ec9 (0x4dd2 bytes) 000761 1713296870.931862 read 0x00c021 thru 0x01570c (0x96ec bytes) 000762 1713296870.934209 trunc from 0x038732 to 0x03e71c (0x5feb bytes) 000763 1713296870.938495 read 0x02845c thru 0x037e5d (0xfa02 bytes) 000764 1713296870.941357 read 0x02257e thru 0x022b1c (0x59f bytes) 000765 1713296870.942761 MIRROR_EXTEND to 8 mirrors 000766 1713296871.086188 fallocate 0x026b3f thru 0x02b665 (0x4b27 bytes) 000767 1713296871.096237 punch from 0x0286f9 to 0x031688 (0x8f90 bytes) 000768 1713296871.097931 mapread 0x009c4d thru 0x00ecfe (0x50b2 bytes) 000771 1713296871.103896 read 0x01456b thru 0x01a92b (0x63c1 bytes) 000772 1713296871.106128 read 0x0279c0 thru 0x0315ea (0x9c2b bytes) 000773 1713296871.108826 punch from 0x000354 to 0x009e6b (0x9b18 bytes) 000774 1713296871.110198 trunc from 0x03e71d to 0x0238c0 (0x1ae5c bytes) 000775 1713296871.116105 punch from 0x004016 to 0x00c612 (0x85fd bytes) 000776 1713296871.117579 write 0x00478f thru 0x012882 (0xe0f4 bytes) 000778 1713296871.119803 trunc from 0x0238c1 to 0x01e6a5 (0x521b bytes) 000779 1713296871.123331 write 0x02070f thru 0x02c8ad (0xc19f bytes) 000780 1713296871.124127 mapread 0x02a3d9 thru 0x02c8ad (0x24d5 bytes) 000781 1713296871.124595 fallocate 0x03c409 thru 0x03ffff (0x3bf7 bytes) 000782 1713296871.126114 write 0x012cac thru 0x0135a9 (0x8fe bytes) 000783 1713296871.126711 punch from 0x00d1d0 to 0x014827 (0x7658 bytes) 000784 1713296871.130565 write 0x01cf12 thru 0x021ed0 (0x4fbf bytes) 000785 1713296871.132418 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000787 1713296871.182057 MIRROR_EXTEND to 8 mirrors 000789 1713296871.294080 punch from 0x00b47d to 0x017da9 (0xc92d bytes) 000791 1713296871.304939 punch from 0x030db5 to 0x03ffff (0xf24b bytes) 000792 1713296871.306657 trunc from 0x040000 to 0x00fa96 (0x30569 bytes) 000795 1713296871.316242 punch from 0x00b24d to 0x00fa96 (0x484a bytes) 000797 1713296871.317994 mapwrite 0x03fe86 thru 0x03ffff (0x17a bytes) 000798 1713296871.327247 trunc from 0x040000 to 0x001945 (0x3e6ba bytes) 000799 1713296871.330759 punch from 0x000493 to 0x001945 (0x14b3 bytes) 000800 1713296871.332441 write 0x03fa53 thru 0x03ffff (0x5ad bytes) 000802 1713296871.333327 fallocate 0x00a2a9 thru 0x011574 (0x72cc bytes) 000803 1713296871.334846 punch from 0x006d86 to 0x00f38f (0x860a bytes) 000804 1713296871.336115 punch from 0x039225 to 0x03ffff (0x6ddb bytes) 000805 1713296871.338610 mapread 0x033c0a thru 0x03d73f (0x9b36 bytes) 000806 1713296871.343650 mapread 0x0174ec thru 0x027009 (0xfb1e bytes) 000807 1713296871.347989 write 0x035d3c thru 0x03b6b6 (0x597b bytes) 000809 1713296871.350791 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000810 1713296871.404828 fallocate 0x0384bf thru 0x03a984 (0x24c6 bytes) 000811 1713296871.406549 mapwrite 0x030d1a thru 0x03da82 (0xcd69 bytes) 000812 1713296871.421076 trunc from 0x040000 to 0x01a213 (0x25dec bytes) 000813 1713296871.426025 trunc from 0x01a214 to 0x01fb26 (0x5913 bytes) 000815 1713296871.429672 punch from 0x008595 to 0x00ece0 (0x674c bytes) 000816 1713296871.430883 read 0x00bbac thru 0x00e991 (0x2de6 bytes) 000817 1713296871.432612 trunc from 0x01fb27 to 0x03418f (0x14669 bytes) 000818 1713296871.436206 mapwrite 0x028bf9 thru 0x033e01 (0xb209 bytes) 000819 1713296871.445864 read 0x011978 thru 0x01f104 (0xd78d bytes) 000822 1713296871.449067 read 0x0293a4 thru 0x02c607 (0x3264 bytes) 000823 1713296871.449405 trunc from 0x034190 to 0x00828f (0x2bf00 bytes) 000824 1713296871.453305 mapread 0x004a2b thru 0x00828f (0x3865 bytes) 000825 1713296871.453784 mapread 0x0001cd thru 0x00828f (0x80c3 bytes) 000827 1713296871.454822 write 0x010e74 thru 0x013b14 (0x2ca1 bytes) 000828 1713296871.455660 read 0x013007 thru 0x013b14 (0xb0e bytes) 000829 1713296871.456026 punch from 0x00717d to 0x013b14 (0xc998 bytes) 000830 1713296871.460212 MIRROR_RESYNC 7 mirrors 000832 1713296871.633656 mapwrite 0x02f08f thru 0x0391e5 (0xa157 bytes) 000833 1713296871.656043 punch from 0x013821 to 0x01fe67 (0xc647 bytes) 000835 1713296871.657828 read 0x02a4eb thru 0x034035 (0x9b4b bytes) 000836 1713296871.658175 fallocate 0x003233 thru 0x00cb58 (0x9926 bytes) 000838 1713296871.660007 write 0x00f1c0 thru 0x014443 (0x5284 bytes) 000841 1713296871.663332 mapread 0x02c80b thru 0x033bfd (0x73f3 bytes) 000844 1713296871.664238 read 0x02b7d0 thru 0x036fab (0xb7dc bytes) 000845 1713296871.664597 write 0x01ec7b thru 0x021653 (0x29d9 bytes) 000846 1713296871.667430 fallocate 0x0341b9 thru 0x037076 (0x2ebe bytes) 000848 1713296871.669291 write 0x030a0a thru 0x036806 (0x5dfd bytes) 000849 1713296871.669992 mapwrite 0x025616 thru 0x028320 (0x2d0b bytes) 000850 1713296871.675574 fallocate 0x03fdef thru 0x03ffff (0x211 bytes) 000851 1713296871.677192 punch from 0x0065ea to 0x010a7b (0xa492 bytes) 000852 1713296871.680216 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000854 1713296871.728234 trunc from 0x0391e6 to 0x026db3 (0x12432 bytes) 000855 1713296871.733383 fallocate 0x022113 thru 0x028d09 (0x6bf7 bytes) 000856 1713296871.736139 MIRROR_EXTEND to 7 mirrors 000857 1713296871.859042 trunc from 0x026db4 to 0x02c864 (0x5ab1 bytes) 000858 1713296871.874524 mapread 0x008f89 thru 0x013e46 (0xaebe bytes) 000859 1713296871.881002 write 0x012e07 thru 0x018104 (0x52fe bytes) 000860 1713296871.881868 punch from 0x00b793 to 0x017ee5 (0xc753 bytes) 000861 1713296871.886183 fallocate 0x00dab5 thru 0x012dca (0x5316 bytes) 000862 1713296871.887745 mapread 0x00a1fd thru 0x00dcee (0x3af2 bytes) 000863 1713296871.890300 fallocate 0x029718 thru 0x02b972 (0x225b bytes) 000864 1713296871.892005 fallocate 0x00b830 thru 0x01b0ea (0xf8bb bytes) 000865 1713296871.893643 punch from 0x013fe2 to 0x023e9e (0xfebd bytes) 000866 1713296871.895316 trunc from 0x02c865 to 0x02ca85 (0x221 bytes) 000868 1713296871.900872 mapread 0x00e79f thru 0x01d898 (0xf0fa bytes) 000869 1713296871.905091 mapread 0x012d66 thru 0x01807f (0x531a bytes) 000870 1713296871.905716 mapread 0x0094da thru 0x01264a (0x9171 bytes) 000871 1713296871.906263 read 0x028b1f thru 0x02ca85 (0x3f67 bytes) 000873 1713296871.908820 mapread 0x01f552 thru 0x02ca85 (0xd534 bytes) 000874 1713296871.910521 punch from 0x00eecf to 0x0194fa (0xa62c bytes) 000875 1713296871.912531 mapread 0x017ad1 thru 0x01e647 (0x6b77 bytes) 000877 1713296871.915725 read 0x024a5f thru 0x02ca85 (0x8027 bytes) 000879 1713296871.916586 punch from 0x01880b to 0x026584 (0xdd7a bytes) 000881 1713296871.918945 read 0x0053c8 thru 0x00baf2 (0x672b bytes) 000882 1713296871.919400 read 0x004cd9 thru 0x00b862 (0x6b8a bytes) 000883 1713296871.919804 write 0x03d110 thru 0x03ffff (0x2ef0 bytes) 000884 1713296871.922544 MIRROR_RESYNC 7 mirrors 000885 1713296872.241315 mapwrite 0x002781 thru 0x0086ad (0x5f2d bytes) 000887 1713296872.272435 write 0x0212a7 thru 0x024146 (0x2ea0 bytes) 000888 1713296872.276105 trunc from 0x040000 to 0x030bde (0xf421 bytes) 000890 1713296872.281740 punch from 0x02d95d to 0x030bde (0x3282 bytes) 000892 1713296872.283609 punch from 0x0071b6 to 0x01506b (0xdeb6 bytes) 000895 1713296872.285996 mapread 0x0276a1 thru 0x02d1f6 (0x5b56 bytes) 000896 1713296872.288720 mapread 0x00143c thru 0x00f775 (0xe33a bytes) 000897 1713296872.292569 trunc from 0x030bdf to 0x02182e (0xf3b0 bytes) 000902 1713296872.297334 mapwrite 0x032035 thru 0x038614 (0x65e0 bytes) 000903 1713296872.309294 trunc from 0x038615 to 0x00c3b3 (0x2c261 bytes) 000904 1713296872.313028 fallocate 0x03f3ca thru 0x03ffff (0xc36 bytes) 000906 1713296872.314973 fallocate 0x03f4a1 thru 0x03ffff (0xb5f bytes) 000907 1713296872.316263 mapread 0x022eb8 thru 0x028490 (0x55d9 bytes) 000908 1713296872.318733 fallocate 0x022669 thru 0x0251fc (0x2b94 bytes) 000909 1713296872.320167 mapwrite 0x011f91 thru 0x0152b2 (0x3322 bytes) 000910 1713296872.328561 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000911 1713296872.375891 trunc from 0x040000 to 0x03a7bd (0x5842 bytes) 000912 1713296872.380394 punch from 0x022e06 to 0x031683 (0xe87e bytes) 000913 1713296872.383564 MIRROR_EXTEND to 7 mirrors 000914 1713296872.513388 mapread 0x030e2a thru 0x038593 (0x776a bytes) 000915 1713296872.522668 punch from 0x010b64 to 0x0148a7 (0x3d44 bytes) 000916 1713296872.534958 fallocate 0x019ce7 thru 0x01d403 (0x371d bytes) 000917 1713296872.537104 mapread 0x02d72d thru 0x039078 (0xb94c bytes) 000918 1713296872.542202 fallocate 0x03447a thru 0x0382c5 (0x3e4c bytes) 000919 1713296872.543787 read 0x007b8f thru 0x010c4f (0x90c1 bytes) 000920 1713296872.546140 mapread 0x02aaa3 thru 0x02c169 (0x16c7 bytes) 000921 1713296872.546598 read 0x01f751 thru 0x026fa2 (0x7852 bytes) 000925 1713296872.549956 fallocate 0x02d8a5 thru 0x037baf (0xa30b bytes) 000926 1713296872.551550 mapwrite 0x006300 thru 0x00be8a (0x5b8b bytes) 000928 1713296872.557541 trunc from 0x03a7be to 0x027a63 (0x12d5a bytes) 000929 1713296872.563579 read 0x0192ca thru 0x021bb6 (0x88ed bytes) 000930 1713296872.563963 punch from 0x020806 to 0x025e1e (0x5619 bytes) 000931 1713296872.567104 MIRROR_RESYNC 7 mirrors 000932 1713296872.833294 fallocate 0x003e19 thru 0x00ac30 (0x6e18 bytes) 000933 1713296872.841047 trunc from 0x027a64 to 0x01ed9c (0x8cc7 bytes) 000934 1713296872.848963 read 0x00dace thru 0x00e7d5 (0xd08 bytes) 000935 1713296872.850596 punch from 0x014b36 to 0x01d11d (0x85e8 bytes) 000936 1713296872.852093 punch from 0x017566 to 0x01ed9c (0x7837 bytes) 000938 1713296872.853564 read 0x01dd0e thru 0x01ed9c (0x108f bytes) 000939 1713296872.856100 write 0x035855 thru 0x03ffff (0xa7ab bytes) 000940 1713296872.856877 read 0x0267be thru 0x03571f (0xef62 bytes) 000941 1713296872.859023 mapwrite 0x03a4b0 thru 0x03a53c (0x8d bytes) 000942 1713296872.864162 read 0x02a9e3 thru 0x039612 (0xec30 bytes) 000943 1713296872.864651 write 0x037402 thru 0x03ffff (0x8bfe bytes) 000944 1713296872.865586 fallocate 0x01a18e thru 0x021531 (0x73a4 bytes) 000946 1713296872.867383 read 0x00e2b0 thru 0x00e779 (0x4ca bytes) 000947 1713296872.867696 mapread 0x034afa thru 0x03d475 (0x897c bytes) 000949 1713296872.868449 read 0x03468c thru 0x03ffff (0xb974 bytes) 000950 1713296872.868731 write 0x03621a thru 0x03bebb (0x5ca2 bytes) 000951 1713296872.869254 fallocate 0x00f675 thru 0x01e1a5 (0xeb31 bytes) 000952 1713296872.870855 read 0x03f031 thru 0x03ffff (0xfcf bytes) 000953 1713296872.871377 write 0x013daf thru 0x01a912 (0x6b64 bytes) 000954 1713296872.872315 read 0x00f968 thru 0x017c36 (0x82cf bytes) 000955 1713296872.872851 fallocate 0x029aa9 thru 0x037a76 (0xdfce bytes) 000960 1713296872.875337 fallocate 0x0102df thru 0x01da9a (0xd7bc bytes) 000961 1713296872.876694 punch from 0x0033c4 to 0x011f0f (0xeb4c bytes) 000963 1713296872.878336 punch from 0x00a1c0 to 0x0142a6 (0xa0e7 bytes) 000964 1713296872.881529 mapread 0x02d5d6 thru 0x039d93 (0xc7be bytes) 000965 1713296872.882009 read 0x032298 thru 0x036c32 (0x499b bytes) 000966 1713296872.882267 mapwrite 0x01cadf thru 0x022f42 (0x6464 bytes) 000967 1713296872.887985 write 0x00381c thru 0x00ed73 (0xb558 bytes) 000968 1713296872.890612 mapread 0x0010e7 thru 0x0080fe (0x7018 bytes) 000969 1713296872.891049 mapread 0x00e9ee thru 0x01bfed (0xd600 bytes) 000971 1713296872.892814 MIRROR_RESYNC 7 mirrors 000972 1713296873.161083 write 0x00406a thru 0x010eb5 (0xce4c bytes) 000973 1713296873.172560 punch from 0x033445 to 0x03a1eb (0x6da7 bytes) 000974 1713296873.174265 trunc from 0x040000 to 0x00b800 (0x347ff bytes) 000976 1713296873.179493 mapread 0x001e4e thru 0x00a780 (0x8933 bytes) 000977 1713296873.181170 punch from 0x009e0f to 0x00b800 (0x19f2 bytes) 000978 1713296873.184207 trunc from 0x00b801 to 0x00844e (0x33b2 bytes) 000979 1713296873.187806 mapread 0x00414a thru 0x00844e (0x4305 bytes) 000980 1713296873.188403 mapwrite 0x034b79 thru 0x03cf44 (0x83cc bytes) 000982 1713296873.202255 mapwrite 0x021330 thru 0x02512a (0x3dfb bytes) 000983 1713296873.210610 MIRROR_EXTEND to 8 mirrors 000984 1713296873.321437 mapwrite 0x00cd2a thru 0x016268 (0x953f bytes) 000985 1713296873.353306 write 0x0297f4 thru 0x02e58b (0x4d98 bytes) 000986 1713296873.357428 read 0x011be1 thru 0x0124b5 (0x8d5 bytes) 000987 1713296873.357729 fallocate 0x0357a7 thru 0x03bd77 (0x65d1 bytes) 000988 1713296873.359072 punch from 0x0379cd to 0x03cf44 (0x5578 bytes) 000989 1713296873.360364 mapread 0x010009 thru 0x0102ac (0x2a4 bytes) 000990 1713296873.361966 MIRROR_EXTEND to 9 mirrors 000991 1713296873.490768 mapread 0x01c8ac thru 0x025232 (0x8987 bytes) 000993 1713296873.500519 MIRROR_EXTEND to 10 mirrors 000994 1713296873.623000 trunc from 0x03cf45 to 0x021f36 (0x1b00e bytes) 000995 1713296873.639694 MIRROR_RESYNC 10 mirrors 000996 1713296873.997271 read 0x003750 thru 0x012a2b (0xf2dc bytes) 000999 1713296873.999635 write 0x006583 thru 0x011d99 (0xb817 bytes) All operations completed A-OK! PASS 70a (15s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 15:47:56 (1713296876) Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre2 Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7925056 .. Extending file size to 4507168 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 4961856 .. Extending file size to 7092672 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7927104 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8267488 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7846656 .. Extending file size to 8802400 .. Extending file size to 8645984 .. Extending file size to 9368320 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9325056 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 12522 12523 12525 12526 12529 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) 192.168.201.146@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg146-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:49:01 (1713296941) Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre2 Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 969600 .. Extending file size to 5341088 .. Extending file size to 6739264 .. Extending file size to 7957504 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8920832 .. Extending file size to 8817440 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9107264 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9041376 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 16039 16040 16042 16044 16045 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) 192.168.201.146@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg146-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:50:05 (1713297005) Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2824 1284864 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3100 1284588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22592 3584428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14688 3592332 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37280 7176760 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2824 1284864 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3100 1284588 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 22592 3584428 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14688 3592332 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37280 7176760 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0341858 s, 61.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0504662 s, 41.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:50:11 (1713297011) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0260295 s, 80.6 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 15:50:15 (1713297015) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } PASS 203 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:50:18 (1713297018) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0188374 s, 111 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xfae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xfae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xfaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:50:21 (1713297021) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0219186 s, 95.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:50:25 (1713297025) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0245797 s, 85.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd2:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:50:49 (1713297049) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138855 s, 75.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 204d (32s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:51:23 (1713297083) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.022412 s, 93.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:51:57 (1713297117) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg146-server: Pool lustre.test_204f created Adding targets to pool oleg146-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg146-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0203801 s, 103 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg146-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg146-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg146-server: Pool lustre.test_204f destroyed PASS 204f (45s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 15:52:44 (1713297164) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:52:47 (1713297167) oleg146-server: Pool lustre.test_206 created Adding targets to pool oleg146-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg146-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg146-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg146-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg146-server: Pool lustre.test_206 destroyed PASS 206 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:53:00 (1713297180) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178787 s, 58.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112727 s, 93.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x14ff34 Mirror 3: 0x14ff34 PASS 207 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:53:04 (1713297184) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:53:07 (1713297187) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:53:09 (1713297189) Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00852077 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1740000 Read this (4 bytes): BBBB 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:53:25 (1713297205) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0024817 s, 3.3 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 15:53:29 (1713297209) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152223 s, 68.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc2 lmm_fid: [0x200000405:0xc2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc2 lmm_fid: [0x200000405:0xc2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe0:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611330 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-16 15:53:29.000000000 -0400 Modify: 2024-04-16 15:53:29.000000000 -0400 Change: 2024-04-16 15:53:30.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:53:33 (1713297213) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107548 s, 97.5 MB/s oleg146-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg146-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg146-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg146-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg146-server: O/280000401/d1/e1: File not found by ext2_lookup PASS 210b (8s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 776 sec ============ 15:53:42 (1713297222) === sanity-flr: start cleanup 15:53:42 (1713297222) === === sanity-flr: finish cleanup 15:53:43 (1713297223) ===