-----============= acceptance-small: sanity-flr ============----- Thu Apr 18 03:23:31 EDT 2024 excepting tests: 6 201 44c === sanity-flr: start setup 03:23:34 (1713425014) === oleg303-client.virtnet: executing check_config_client /mnt/lustre oleg303-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg303-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012e1c6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012e1c6000.idle_timeout=debug disable quota as required oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 03:23:41 (1713425021) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7507] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 03:23:43 (1713425023) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:23:48 (1713425028) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:23:50 (1713425030) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 03:23:52 (1713425032) PASS 0d (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:23:59 (1713425039) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:24:01 (1713425041) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 03:24:03 (1713425043) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012e1c6000.stats=clear osc.lustre-OST0001-osc-ffff88012e1c6000.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:24:07 (1713425047) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:24:11 (1713425051) PASS 0j (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:24:14 (1713425054) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 03:24:17 (1713425057) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 03:24:20 (1713425060) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:24:23 (1713425063) PASS 3 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:24:27 (1713425067) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:24:30 (1713425070) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 03:24:33 (1713425073) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:24:36 (1713425076) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.398655 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0238517 s, 44.0 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:24:40 (1713425080) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0424627 s, 148 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:24:43 (1713425083) Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e1c6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e1c6000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e1c6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e1c6000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:25:07 (1713425107) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.114765 s, 73.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e1c6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e1c6000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:25:17 (1713425117) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e1c6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e1c6000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e1c6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e1c6000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:25:37 (1713425137) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.853083 s, 61.5 MB/s -rw-r--r-- 1 root root 52428800 Apr 18 03:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.214s user 0m0.000s sys 0m0.103s Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e1c6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e1c6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.198s user 0m0.000s sys 0m0.099s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e1c6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e1c6000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.225s user 0m0.000s sys 0m0.113s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012e1c6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e1c6000.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:25:58 (1713425158) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:26:00 (1713425160) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:26:03 (1713425163) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 03:26:05 (1713425165) PASS 35 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 03:26:08 (1713425168) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.10977 s, 115 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:26:12 (1713425172) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00856606 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00880603 s, 119 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:26:15 (1713425175) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:26:19 (1713425179) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000560044 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000551719 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000679804 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00058701 s, 1.7 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 03:26:23 (1713425183) create test files with size 15737833 .. 710c3e7e2dbcc1abafdbcc855bd866c6 - ccb5dc2309b70ddc84a4c32937f2c74c - 661b4ffc66a44d4c0ba04ea812fc6167 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre PASS 37 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 03:26:29 (1713425189) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:26:34 (1713425194) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:26:37 (1713425197) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131635 s, 79.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00789266 s, 133 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 03:26:40 (1713425200) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0190369 s, 110 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0788414 s, 53.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0303233 s, 69.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 03:26:44 (1713425204) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 03:26:46 (1713425206) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:26:48 (1713425208) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00815394 s, 129 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 03:26:52 (1713425212) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0755743 s, 41.6 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:26:55 (1713425215) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:26:59 (1713425219) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0656746 s, 160 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:27:03 (1713425223) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800aadd4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0724757 s, 145 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a84a1000.xattr_cache=1 PASS 44e (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:27:07 (1713425227) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 03:27:15 (1713425235) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 03:27:22 (1713425242) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 03:27:24 (1713425244) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.060945 s, 51.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0482289 s, 65.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 7de0cf21905d62be787f6d30537271a7 - mirror 1 checksum: 31d90e2f3731173480a23bd4c1b43233 - mirror 2 checksum: 7de0cf21905d62be787f6d30537271a7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 03:27:25 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 03:27:25 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7de0cf21905d62be787f6d30537271a7 - mirror 1 checksum: 31d90e2f3731173480a23bd4c1b43233 - mirror 2 checksum: 31d90e2f3731173480a23bd4c1b43233 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 03:27:25 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 03:27:25 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 03:27:28 (1713425248) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00825846 s, 127 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:27:31 (1713425251) Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server 03:27:38 (1713425258) shut down Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 03:27:51 (1713425271) targets are mounted 03:27:51 (1713425271) facet_failover done oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00888769 s, 118 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 50A (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:28:00 (1713425280) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00727625 s, 2.8 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00333313 s, 6.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 03:28:04 (1713425284) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00821384 s, 2.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00259702 s, 7.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:28:07 (1713425287) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00385681 s, 4.2 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 03:28:11 (1713425291) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00792672 s, 2.6 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:28:15 (1713425295) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00149553 s, 2.7 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00622926 s, 658 kB/s PASS 60a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:28:18 (1713425298) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:28:25 (1713425305) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:28:44 (1713425324) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:29:03 (1713425343) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:29:22 (1713425362) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 03:29:25 (1713425365) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 30493 mirror split pid 30494 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 30493 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 30494 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 03:30:29 (1713425429) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 5429 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1713425429.555606 write 0x02ad20 thru 0x039c7f (0xef60 bytes) fallocating to largest ever: 0x264a0 000002 1713425429.559802 fallocate 0x024bbe thru 0x02649f (0x18e2 bytes) 000003 1713425429.561345 write 0x0297be thru 0x0321b1 (0x89f4 bytes) 000006 1713425429.563281 write 0x01c27a thru 0x02b8d6 (0xf65d bytes) 000007 1713425429.565181 mapread 0x024377 thru 0x026938 (0x25c2 bytes) 000008 1713425429.565654 write 0x010763 thru 0x01e8a8 (0xe146 bytes) 000010 1713425429.568926 write 0x0297ab thru 0x02a723 (0xf79 bytes) 000011 1713425429.569548 mapread 0x0243f4 thru 0x026fc8 (0x2bd5 bytes) 000012 1713425429.570091 mapread 0x016940 thru 0x01af4d (0x460e bytes) 000013 1713425429.570669 write 0x00408c thru 0x004f6e (0xee3 bytes) 000014 1713425429.572858 trunc from 0x039c80 to 0x008b53 (0x3112c bytes) 000018 1713425429.577362 punch from 0x00256b to 0x0034be (0xf54 bytes) 000019 1713425429.578523 mapread 0x007edc thru 0x008b53 (0xc78 bytes) 000020 1713425429.578898 write 0x03b671 thru 0x03bdc6 (0x756 bytes) 000021 1713425429.579388 read 0x03ab4d thru 0x03ac20 (0xd4 bytes) 000022 1713425429.580741 write 0x01db72 thru 0x02c2ea (0xe779 bytes) 000024 1713425429.584073 write 0x026e96 thru 0x02a297 (0x3402 bytes) 000028 1713425429.585446 mapread 0x038372 thru 0x0399d7 (0x1666 bytes) 000030 1713425429.587684 fallocate 0x016426 thru 0x02525b (0xee36 bytes) 000031 1713425429.589387 write 0x0055e1 thru 0x012d33 (0xd753 bytes) 000032 1713425429.591424 write 0x005887 thru 0x00af27 (0x56a1 bytes) 000034 1713425429.592555 mapwrite 0x008c71 thru 0x018869 (0xfbf9 bytes) 000035 1713425429.603252 punch from 0x0236d3 to 0x02b035 (0x7963 bytes) 000037 1713425429.605099 write 0x001820 thru 0x00b9a9 (0xa18a bytes) 000038 1713425429.605790 write 0x03d077 thru 0x03ffff (0x2f89 bytes) 000039 1713425429.606221 trunc from 0x040000 to 0x0102f5 (0x2fd0a bytes) 000041 1713425429.611257 punch from 0x005850 to 0x00a5b5 (0x4d66 bytes) 000043 1713425429.615283 mapread 0x005bf4 thru 0x00ba7e (0x5e8b bytes) truncating to largest ever: 0x37fd1 000044 1713425429.617453 trunc from 0x0102f6 to 0x037fd0 (0x27cdb bytes) 000045 1713425429.621240 mapread 0x030329 thru 0x037fd0 (0x7ca8 bytes) 000046 1713425429.623977 read 0x0068f9 thru 0x0125e3 (0xbceb bytes) 000047 1713425429.626646 read 0x02334d thru 0x02e6cb (0xb37f bytes) 000050 1713425429.627446 fallocate 0x0235de thru 0x02b4c3 (0x7ee6 bytes) 000051 1713425429.628914 mapread 0x011d89 thru 0x01371b (0x1993 bytes) 000054 1713425429.629777 mapread 0x030318 thru 0x037fd0 (0x7cb9 bytes) 000056 1713425429.630585 read 0x013429 thru 0x0214c1 (0xe099 bytes) 000057 1713425429.630969 read 0x00c051 thru 0x00c93a (0x8ea bytes) 000058 1713425429.631237 mapread 0x0042a5 thru 0x00f21e (0xaf7a bytes) 000059 1713425429.631640 punch from 0x000ab0 to 0x00b201 (0xa752 bytes) 000060 1713425429.633240 write 0x00178e thru 0x006f7e (0x57f1 bytes) 000062 1713425429.636061 mapwrite 0x007da6 thru 0x012e31 (0xb08c bytes) 000063 1713425429.643168 mapread 0x00b51b thru 0x01177e (0x6264 bytes) truncating to largest ever: 0x3dc49 000065 1713425429.644609 trunc from 0x037fd1 to 0x03dc48 (0x5c78 bytes) fallocating to largest ever: 0x40000 000066 1713425429.648407 fallocate 0x03fedc thru 0x03ffff (0x124 bytes) 000067 1713425429.649823 fallocate 0x03a3c1 thru 0x03ffff (0x5c3f bytes) 000068 1713425429.651056 trunc from 0x040000 to 0x035f63 (0xa09c bytes) 000069 1713425429.654592 punch from 0x02fbfe to 0x035f63 (0x6366 bytes) 000070 1713425429.656005 write 0x03e876 thru 0x03ffff (0x178a bytes) 000071 1713425429.656725 write 0x0217c4 thru 0x026ed7 (0x5714 bytes) 000074 1713425429.658108 punch from 0x008ea6 to 0x0142e6 (0xb441 bytes) 000075 1713425429.659623 mapwrite 0x011194 thru 0x01e94b (0xd7b8 bytes) 000078 1713425429.672491 mapread 0x00e8ac thru 0x0199e0 (0xb135 bytes) 000079 1713425429.676025 read 0x01dc64 thru 0x01f162 (0x14ff bytes) 000081 1713425429.676580 punch from 0x013a29 to 0x01a543 (0x6b1b bytes) 000084 1713425429.679014 read 0x00fb86 thru 0x0101d0 (0x64b bytes) 000085 1713425429.679493 fallocate 0x01d217 thru 0x025bc5 (0x89af bytes) 000086 1713425429.681110 punch from 0x023aea to 0x029c54 (0x616b bytes) 000087 1713425429.684019 MIRROR_EXTEND to 2 mirrors 000088 1713425429.803543 trunc from 0x040000 to 0x02f262 (0x10d9d bytes) 000090 1713425429.816264 mapwrite 0x00d119 thru 0x013ffa (0x6ee2 bytes) 000092 1713425429.828068 MIRROR_EXTEND to 3 mirrors 000095 1713425429.950633 mapread 0x0068ac thru 0x0130a0 (0xc7f5 bytes) 000096 1713425429.959455 punch from 0x009998 to 0x00a5f1 (0xc5a bytes) 000098 1713425429.967819 punch from 0x00d1a1 to 0x014430 (0x7290 bytes) 000099 1713425429.969474 read 0x0228bc thru 0x02b6a6 (0x8deb bytes) 000100 1713425429.971904 fallocate 0x015a90 thru 0x023a92 (0xe003 bytes) 000102 1713425429.973633 read 0x005fa1 thru 0x00f7ac (0x980c bytes) 000103 1713425429.975936 punch from 0x01d936 to 0x021afe (0x41c9 bytes) 000104 1713425429.977283 trunc from 0x02f263 to 0x009b38 (0x2572a bytes) 000106 1713425429.987044 fallocate 0x00c29f thru 0x015042 (0x8da4 bytes) 000107 1713425429.989764 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000108 1713425430.032642 fallocate 0x02957d thru 0x02a713 (0x1197 bytes) 000109 1713425430.034711 punch from 0x0049d9 to 0x012e3a (0xe462 bytes) 000110 1713425430.036169 fallocate 0x01ada7 thru 0x01c6c3 (0x191d bytes) 000111 1713425430.038554 MIRROR_RESYNC 2 mirrors 000112 1713425430.134070 trunc from 0x015043 to 0x03674d (0x2170b bytes) 000113 1713425430.143054 read 0x012455 thru 0x021141 (0xeced bytes) 000114 1713425430.145707 mapwrite 0x023fd0 thru 0x026327 (0x2358 bytes) 000117 1713425430.156255 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000119 1713425430.183991 punch from 0x02d05c to 0x03674d (0x96f2 bytes) 000122 1713425430.186440 mapwrite 0x00efcb thru 0x013bc5 (0x4bfb bytes) 000123 1713425430.194945 read 0x01cab7 thru 0x029268 (0xc7b2 bytes) 000125 1713425430.197649 mapwrite 0x022e25 thru 0x0312e2 (0xe4be bytes) 000126 1713425430.205235 mapread 0x02a63c thru 0x02eda4 (0x4769 bytes) 000130 1713425430.206530 fallocate 0x0146de thru 0x01a06b (0x598e bytes) 000131 1713425430.208374 punch from 0x00ab24 to 0x00bfc0 (0x149d bytes) 000132 1713425430.209641 read 0x0078a4 thru 0x01098b (0x90e8 bytes) 000133 1713425430.212457 punch from 0x02d2b3 to 0x03674d (0x949b bytes) 000134 1713425430.214070 trunc from 0x03674e to 0x038363 (0x1c16 bytes) 000136 1713425430.218852 mapread 0x00eb8b thru 0x01d932 (0xeda8 bytes) 000139 1713425430.220250 mapread 0x030390 thru 0x037b8f (0x7800 bytes) 000140 1713425430.229607 mapread 0x00a997 thru 0x01754a (0xcbb4 bytes) 000141 1713425430.230294 trunc from 0x038364 to 0x0005d3 (0x37d90 bytes) 000142 1713425430.234309 write 0x0150cd thru 0x0198f7 (0x482b bytes) 000145 1713425430.235436 write 0x018300 thru 0x01c4ac (0x41ad bytes) 000146 1713425430.235907 fallocate 0x006961 thru 0x00cbff (0x629f bytes) 000147 1713425430.237404 mapwrite 0x02b431 thru 0x02d8f9 (0x24c9 bytes) 000148 1713425430.249964 trunc from 0x02d8fa to 0x03ce5f (0xf566 bytes) 000149 1713425430.253507 punch from 0x0071b3 to 0x013715 (0xc563 bytes) 000150 1713425430.254806 punch from 0x03c093 to 0x03ce5f (0xdcd bytes) 000151 1713425430.256233 mapread 0x024171 thru 0x02bbeb (0x7a7b bytes) 000152 1713425430.264548 mapread 0x018617 thru 0x02064c (0x8036 bytes) 000154 1713425430.269506 mapread 0x033f1a thru 0x03ce5f (0x8f46 bytes) 000155 1713425430.276384 punch from 0x0353c1 to 0x03ce5f (0x7a9f bytes) 000156 1713425430.277720 trunc from 0x03ce60 to 0x0206ef (0x1c770 bytes) 000157 1713425430.281862 write 0x02ede2 thru 0x03b5b0 (0xc7cf bytes) 000158 1713425430.282993 mapwrite 0x02e104 thru 0x03b2be (0xd1bb bytes) 000159 1713425430.289266 punch from 0x018c27 to 0x025367 (0xc741 bytes) 000161 1713425430.290870 fallocate 0x0179ff thru 0x0180d8 (0x6da bytes) 000162 1713425430.292086 mapwrite 0x01f102 thru 0x02d922 (0xe821 bytes) 000163 1713425430.313856 read 0x01e841 thru 0x02ad8a (0xc54a bytes) 000164 1713425430.318534 mapread 0x01f5d7 thru 0x029fcc (0xa9f6 bytes) 000168 1713425430.321311 MIRROR_EXTEND to 2 mirrors 000170 1713425430.436862 punch from 0x026cd2 to 0x02ef7a (0x82a9 bytes) 000171 1713425430.444625 read 0x02311f thru 0x031578 (0xe45a bytes) 000173 1713425430.447177 mapwrite 0x03ad5d thru 0x03ffff (0x52a3 bytes) 000174 1713425430.465166 mapread 0x029bbc thru 0x03231c (0x8761 bytes) 000179 1713425430.467987 read 0x007169 thru 0x009fc1 (0x2e59 bytes) 000182 1713425430.471166 read 0x01276a thru 0x0158ec (0x3183 bytes) 000183 1713425430.471506 mapread 0x003187 thru 0x00f0be (0xbf38 bytes) 000184 1713425430.472007 read 0x0395fc thru 0x03e900 (0x5305 bytes) 000185 1713425430.472342 mapwrite 0x032e3c thru 0x038c30 (0x5df5 bytes) 000186 1713425430.477135 mapwrite 0x031642 thru 0x03d120 (0xbadf bytes) 000187 1713425430.484016 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000188 1713425430.511930 trunc from 0x040000 to 0x02d293 (0x12d6c bytes) 000189 1713425430.516538 trunc from 0x02d294 to 0x0141ad (0x190e6 bytes) 000190 1713425430.519958 punch from 0x01114c to 0x0141ad (0x3062 bytes) 000191 1713425430.521212 trunc from 0x0141ae to 0x03d6bf (0x29512 bytes) 000192 1713425430.524617 write 0x023624 thru 0x02a587 (0x6f64 bytes) 000193 1713425430.527317 write 0x037f71 thru 0x03ffff (0x808f bytes) 000194 1713425430.529600 mapwrite 0x006790 thru 0x01640f (0xfc80 bytes) 000195 1713425430.540059 punch from 0x0377c4 to 0x03ffff (0x883c bytes) 000196 1713425430.541762 fallocate 0x006fc5 thru 0x013eda (0xcf16 bytes) 000197 1713425430.543176 read 0x01996c thru 0x022527 (0x8bbc bytes) 000198 1713425430.545388 read 0x02b259 thru 0x035b69 (0xa911 bytes) 000199 1713425430.548259 write 0x00b774 thru 0x01a037 (0xe8c4 bytes) 000204 1713425430.551251 mapread 0x01dd3b thru 0x01ef59 (0x121f bytes) 000205 1713425430.551937 read 0x038964 thru 0x03ffff (0x769c bytes) 000208 1713425430.552811 punch from 0x03c59e to 0x03ffff (0x3a62 bytes) 000209 1713425430.554781 read 0x012596 thru 0x0126e0 (0x14b bytes) 000211 1713425430.555833 fallocate 0x018dce thru 0x01dade (0x4d11 bytes) 000212 1713425430.557718 punch from 0x02eeff to 0x03ea90 (0xfb92 bytes) 000213 1713425430.560855 MIRROR_EXTEND to 2 mirrors 000214 1713425430.677781 write 0x024922 thru 0x02de0e (0x94ed bytes) 000215 1713425430.687537 mapwrite 0x035036 thru 0x03ae19 (0x5de4 bytes) 000216 1713425430.695098 write 0x03803c thru 0x03ad5b (0x2d20 bytes) 000218 1713425430.695820 trunc from 0x040000 to 0x0069ba (0x39645 bytes) 000219 1713425430.704213 read 0x002bb4 thru 0x0069ba (0x3e07 bytes) 000221 1713425430.706630 write 0x00bd50 thru 0x01571b (0x99cc bytes) 000223 1713425430.707957 punch from 0x0093cd to 0x00a8bf (0x14f3 bytes) 000224 1713425430.709217 punch from 0x0015ca to 0x00e267 (0xcc9e bytes) 000226 1713425430.712439 fallocate 0x00554b thru 0x00b2ac (0x5d62 bytes) 000227 1713425430.713706 mapwrite 0x02af4d thru 0x02c465 (0x1519 bytes) 000228 1713425430.724344 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000230 1713425430.750379 mapread 0x019f3a thru 0x0267b7 (0xc87e bytes) 000231 1713425430.756645 read 0x01dee6 thru 0x025d3b (0x7e56 bytes) 000232 1713425430.756929 trunc from 0x02c466 to 0x005083 (0x273e2 bytes) 000233 1713425430.761263 read 0x001b09 thru 0x005083 (0x357b bytes) 000235 1713425430.763667 fallocate 0x0144e8 thru 0x01ab34 (0x664d bytes) 000236 1713425430.765136 trunc from 0x005084 to 0x006251 (0x11ce bytes) 000237 1713425430.768784 write 0x035812 thru 0x03af90 (0x577f bytes) 000238 1713425430.769589 punch from 0x02854b to 0x02fd9c (0x7852 bytes) 000239 1713425430.770935 fallocate 0x007439 thru 0x016a9b (0xf663 bytes) 000240 1713425430.772131 mapread 0x030dd1 thru 0x03af90 (0xa1c0 bytes) 000242 1713425430.775043 mapwrite 0x03a9da thru 0x03ffff (0x5626 bytes) 000243 1713425430.786463 MIRROR_EXTEND to 2 mirrors 000244 1713425430.892920 read 0x03fe96 thru 0x03ffff (0x16a bytes) 000246 1713425430.897120 mapwrite 0x0214d4 thru 0x029dd1 (0x88fe bytes) 000247 1713425430.926937 fallocate 0x019c26 thru 0x026fb0 (0xd38b bytes) 000249 1713425430.928529 trunc from 0x040000 to 0x00fe1b (0x301e4 bytes) 000250 1713425430.936844 punch from 0x00a7ca to 0x00e9de (0x4215 bytes) 000251 1713425430.937955 mapwrite 0x01fa3d thru 0x026798 (0x6d5c bytes) 000252 1713425430.948578 mapwrite 0x02bcfe thru 0x03a2aa (0xe5ad bytes) 000254 1713425430.962209 mapwrite 0x02a4b6 thru 0x03628d (0xbdd8 bytes) 000255 1713425430.967919 mapwrite 0x0285f0 thru 0x02d51c (0x4f2d bytes) 000256 1713425430.972620 fallocate 0x02cf91 thru 0x039902 (0xc972 bytes) 000257 1713425430.973847 trunc from 0x03a2ab to 0x02b778 (0xeb32 bytes) 000259 1713425430.977701 mapread 0x00c33b thru 0x00fd64 (0x3a2a bytes) 000260 1713425430.980249 read 0x0164b7 thru 0x021cc3 (0xb80d bytes) 000261 1713425430.981790 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000262 1713425431.009117 mapwrite 0x039792 thru 0x03ffff (0x686e bytes) 000263 1713425431.023125 mapread 0x03be82 thru 0x03e457 (0x25d6 bytes) 000265 1713425431.024000 punch from 0x00fb92 to 0x01aef1 (0xb360 bytes) 000267 1713425431.025550 mapwrite 0x0366a3 thru 0x03d617 (0x6f75 bytes) 000268 1713425431.031274 MIRROR_EXTEND to 2 mirrors 000269 1713425431.171436 mapread 0x01fa08 thru 0x0200e3 (0x6dc bytes) 000270 1713425431.178025 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000271 1713425431.203636 trunc from 0x040000 to 0x000b27 (0x3f4d8 bytes) 000272 1713425431.210120 mapwrite 0x008f6f thru 0x0129d0 (0x9a62 bytes) 000273 1713425431.223703 read 0x00f378 thru 0x0129d0 (0x3659 bytes) 000276 1713425431.225311 trunc from 0x0129d1 to 0x01ff77 (0xd5a7 bytes) 000277 1713425431.229090 trunc from 0x01ff78 to 0x024993 (0x4a1c bytes) 000278 1713425431.233425 MIRROR_EXTEND to 2 mirrors 000279 1713425431.346934 trunc from 0x024994 to 0x03c109 (0x17776 bytes) 000280 1713425431.357389 read 0x019f24 thru 0x029bdc (0xfcb9 bytes) 000282 1713425431.360000 fallocate 0x00197d thru 0x00c1a4 (0xa828 bytes) 000283 1713425431.361308 fallocate 0x0103b5 thru 0x0168a9 (0x64f5 bytes) 000284 1713425431.362687 write 0x007764 thru 0x011c4f (0xa4ec bytes) 000286 1713425431.365559 write 0x02fe8b thru 0x035852 (0x59c8 bytes) 000287 1713425431.367830 read 0x00de3e thru 0x01d1af (0xf372 bytes) 000288 1713425431.370264 mapwrite 0x011017 thru 0x01d414 (0xc3fe bytes) 000289 1713425431.377609 MIRROR_RESYNC 2 mirrors 000290 1713425431.460029 mapwrite 0x02195a thru 0x0293a0 (0x7a47 bytes) 000291 1713425431.489370 write 0x039537 thru 0x03ffff (0x6ac9 bytes) 000292 1713425431.491931 fallocate 0x00ceac thru 0x00ebb9 (0x1d0e bytes) 000293 1713425431.493282 trunc from 0x040000 to 0x010d14 (0x2f2eb bytes) 000294 1713425431.501232 mapwrite 0x0318c7 thru 0x03ffff (0xe739 bytes) 000295 1713425431.514540 mapread 0x01ffcb thru 0x02f0a9 (0xf0df bytes) 000296 1713425431.518510 write 0x024321 thru 0x02c0ea (0x7dca bytes) 000298 1713425431.519341 read 0x01c8af thru 0x026809 (0x9f5b bytes) 000299 1713425431.519674 trunc from 0x040000 to 0x03cc17 (0x33e8 bytes) 000300 1713425431.523114 mapwrite 0x02dc26 thru 0x034661 (0x6a3c bytes) 000301 1713425431.528974 trunc from 0x03cc18 to 0x0020b6 (0x3ab61 bytes) 000302 1713425431.533833 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000304 1713425431.563578 punch from 0x001841 to 0x0020b6 (0x876 bytes) 000305 1713425431.565432 read 0x00059f thru 0x0020b6 (0x1b18 bytes) 000306 1713425431.567118 write 0x0111bd thru 0x015c28 (0x4a6c bytes) 000307 1713425431.569104 fallocate 0x035eb6 thru 0x03ffff (0xa14a bytes) 000308 1713425431.570564 read 0x0149d3 thru 0x016065 (0x1693 bytes) 000309 1713425431.573827 punch from 0x023a97 to 0x027dd3 (0x433d bytes) 000310 1713425431.575159 punch from 0x03802d to 0x03ffff (0x7fd3 bytes) 000311 1713425431.576474 read 0x00bbc2 thru 0x015c8d (0xa0cc bytes) 000312 1713425431.576830 mapread 0x0219ba thru 0x025497 (0x3ade bytes) 000313 1713425431.580274 mapread 0x03f873 thru 0x03ffff (0x78d bytes) 000317 1713425431.582104 trunc from 0x040000 to 0x02ea64 (0x1159b bytes) 000320 1713425431.587038 trunc from 0x02ea65 to 0x00108f (0x2d9d5 bytes) 000321 1713425431.590733 write 0x033c76 thru 0x03ffff (0xc38a bytes) 000322 1713425431.591507 write 0x0068ea thru 0x00f378 (0x8a8f bytes) 000323 1713425431.594269 punch from 0x00fb8e to 0x010792 (0xc05 bytes) 000324 1713425431.597560 punch from 0x0079b7 to 0x010912 (0x8f5c bytes) 000326 1713425431.598936 write 0x037bcc thru 0x03ffff (0x8434 bytes) 000327 1713425431.599496 read 0x008e56 thru 0x010816 (0x79c1 bytes) 000329 1713425431.602661 fallocate 0x018145 thru 0x023295 (0xb151 bytes) 000331 1713425431.604237 mapread 0x014ae4 thru 0x01d012 (0x852f bytes) 000332 1713425431.604688 read 0x027c47 thru 0x030396 (0x8750 bytes) 000333 1713425431.604949 punch from 0x010153 to 0x011ba9 (0x1a57 bytes) 000334 1713425431.606242 mapread 0x00c758 thru 0x015876 (0x911f bytes) 000336 1713425431.608011 punch from 0x01ddaa to 0x020ba8 (0x2dff bytes) 000337 1713425431.609257 trunc from 0x040000 to 0x02ebc9 (0x11436 bytes) 000338 1713425431.613158 trunc from 0x02ebca to 0x02c91e (0x22ab bytes) 000339 1713425431.616688 trunc from 0x02c91f to 0x01c2c8 (0x10656 bytes) 000340 1713425431.620241 mapwrite 0x0204a1 thru 0x0207c6 (0x326 bytes) 000341 1713425431.629083 trunc from 0x0207c7 to 0x0008e4 (0x1fee2 bytes) 000342 1713425431.632749 fallocate 0x03134f thru 0x037a90 (0x6742 bytes) 000343 1713425431.634291 punch from 0x0002c7 to 0x0008e4 (0x61e bytes) 000345 1713425431.636008 write 0x01170b thru 0x01e394 (0xcc8a bytes) 000347 1713425431.637016 mapwrite 0x021382 thru 0x02cab0 (0xb72f bytes) 000348 1713425431.662213 MIRROR_EXTEND to 2 mirrors 000349 1713425431.769200 read 0x01269a thru 0x019b61 (0x74c8 bytes) 000350 1713425431.774223 MIRROR_EXTEND to 3 mirrors 000352 1713425431.878808 read 0x022c46 thru 0x024a0a (0x1dc5 bytes) 000353 1713425431.880393 fallocate 0x02ba5a thru 0x02d2c0 (0x1867 bytes) 000354 1713425431.887625 mapread 0x01f553 thru 0x0222d9 (0x2d87 bytes) 000355 1713425431.891578 trunc from 0x02cab1 to 0x02cff7 (0x547 bytes) 000356 1713425431.901611 MIRROR_EXTEND to 4 mirrors 000357 1713425432.008334 punch from 0x008912 to 0x009cb4 (0x13a3 bytes) 000358 1713425432.017725 read 0x025d4a thru 0x02cff7 (0x72ae bytes) 000359 1713425432.020512 trunc from 0x02cff8 to 0x0158b6 (0x17741 bytes) 000361 1713425432.030947 fallocate 0x012a2c thru 0x020062 (0xd637 bytes) 000362 1713425432.032236 mapread 0x0114a6 thru 0x012492 (0xfed bytes) 000364 1713425432.035271 write 0x005aba thru 0x0085d6 (0x2b1d bytes) 000365 1713425432.037551 mapread 0x003236 thru 0x0076ac (0x4477 bytes) 000366 1713425432.037936 mapread 0x002d37 thru 0x004b85 (0x1e4f bytes) 000367 1713425432.038442 punch from 0x0154ac to 0x0158b6 (0x40b bytes) 000369 1713425432.040075 fallocate 0x017e00 thru 0x01e3ba (0x65bb bytes) 000370 1713425432.041420 trunc from 0x0158b7 to 0x027674 (0x11dbe bytes) 000371 1713425432.045173 fallocate 0x01e168 thru 0x02c68b (0xe524 bytes) 000372 1713425432.047790 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000374 1713425432.097378 mapread 0x001f3b thru 0x011147 (0xf20d bytes) 000375 1713425432.104357 read 0x01ea5d thru 0x027674 (0x8c18 bytes) 000378 1713425432.107139 fallocate 0x00e709 thru 0x01a8ec (0xc1e4 bytes) 000379 1713425432.109714 MIRROR_EXTEND to 4 mirrors 000380 1713425432.221701 trunc from 0x027675 to 0x015b19 (0x11b5b bytes) 000381 1713425432.235462 fallocate 0x03c3b1 thru 0x03ffff (0x3c4f bytes) 000382 1713425432.237119 punch from 0x013669 to 0x015b19 (0x24b1 bytes) 000384 1713425432.240161 MIRROR_RESYNC 4 mirrors 000385 1713425432.410777 fallocate 0x0349a4 thru 0x03aac5 (0x6122 bytes) 000386 1713425432.419022 read 0x01bf05 thru 0x020e46 (0x4f42 bytes) 000387 1713425432.421634 write 0x02bd08 thru 0x03108a (0x5383 bytes) 000388 1713425432.424171 write 0x026786 thru 0x02fa30 (0x92ab bytes) 000389 1713425432.426766 fallocate 0x03609d thru 0x039a91 (0x39f5 bytes) 000390 1713425432.428261 fallocate 0x03ea0f thru 0x03ffff (0x15f1 bytes) 000391 1713425432.430873 MIRROR_EXTEND to 5 mirrors 000392 1713425432.564729 mapread 0x002378 thru 0x012104 (0xfd8d bytes) 000393 1713425432.574138 mapwrite 0x02eac2 thru 0x03a656 (0xbb95 bytes) 000394 1713425432.610335 write 0x0244a6 thru 0x02924d (0x4da8 bytes) 000395 1713425432.613670 mapwrite 0x00a666 thru 0x0115ba (0x6f55 bytes) 000396 1713425432.621056 write 0x01bd2a thru 0x023f5d (0x8234 bytes) 000397 1713425432.623307 mapread 0x03a995 thru 0x03ffff (0x566b bytes) 000398 1713425432.623807 write 0x02a72e thru 0x03514f (0xaa22 bytes) 000400 1713425432.625964 MIRROR_RESYNC 5 mirrors 000401 1713425432.873594 read 0x0115c7 thru 0x01e377 (0xcdb1 bytes) 000402 1713425432.875827 fallocate 0x02864c thru 0x02ea5f (0x6414 bytes) 000403 1713425432.883269 read 0x03baf0 thru 0x03ffff (0x4510 bytes) 000405 1713425432.887759 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000406 1713425432.930583 trunc from 0x040000 to 0x020dae (0x1f251 bytes) 000407 1713425432.940111 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000408 1713425432.979797 fallocate 0x00ce43 thru 0x00fdd7 (0x2f95 bytes) 000410 1713425432.981480 trunc from 0x020daf to 0x03b280 (0x1a4d2 bytes) 000411 1713425432.985714 mapread 0x021f40 thru 0x02e832 (0xc8f3 bytes) 000412 1713425432.991596 read 0x00d45c thru 0x0197c5 (0xc36a bytes) 000413 1713425432.993611 fallocate 0x0341f1 thru 0x03bd7f (0x7b8f bytes) 000414 1713425432.995085 read 0x001c33 thru 0x002b94 (0xf62 bytes) 000415 1713425432.997380 punch from 0x035ced to 0x037099 (0x13ad bytes) 000416 1713425432.998565 write 0x010f52 thru 0x0126cc (0x177b bytes) 000417 1713425432.999146 read 0x00b012 thru 0x0198e2 (0xe8d1 bytes) 000419 1713425432.999716 read 0x01ca8e thru 0x029477 (0xc9ea bytes) 000420 1713425433.000125 trunc from 0x03b281 to 0x002f62 (0x3831e bytes) 000421 1713425433.004054 read 0x00237f thru 0x002f62 (0xbe4 bytes) 000422 1713425433.004463 trunc from 0x002f63 to 0x0038df (0x97d bytes) 000423 1713425433.007972 read 0x00222c thru 0x0038df (0x16b4 bytes) 000425 1713425433.009666 mapread 0x000215 thru 0x000ca0 (0xa8c bytes) 000426 1713425433.010253 punch from 0x00347d to 0x0038df (0x463 bytes) 000427 1713425433.011636 fallocate 0x015898 thru 0x0237d9 (0xdf42 bytes) 000428 1713425433.013348 read 0x0075b2 thru 0x00963e (0x208d bytes) 000429 1713425433.015863 trunc from 0x0237da to 0x00d9a0 (0x15e39 bytes) 000430 1713425433.019436 mapwrite 0x0073cd thru 0x0082ab (0xedf bytes) 000431 1713425433.023968 trunc from 0x00d9a1 to 0x024f1a (0x1757a bytes) 000434 1713425433.027984 mapread 0x01cfe6 thru 0x02345a (0x6475 bytes) 000435 1713425433.036219 mapwrite 0x00ee83 thru 0x01ac86 (0xbe04 bytes) 000436 1713425433.056569 MIRROR_RESYNC 3 mirrors 000437 1713425433.180170 mapread 0x014a2c thru 0x01659c (0x1b71 bytes) 000438 1713425433.184246 read 0x00ec9c thru 0x01c615 (0xd97a bytes) 000439 1713425433.184525 mapwrite 0x025c43 thru 0x02b19f (0x555d bytes) 000441 1713425433.203050 mapwrite 0x02d160 thru 0x0314d4 (0x4375 bytes) 000442 1713425433.213570 fallocate 0x0121f9 thru 0x014948 (0x2750 bytes) 000444 1713425433.215303 punch from 0x01352e to 0x017fa9 (0x4a7c bytes) 000445 1713425433.216600 punch from 0x00bdb7 to 0x01530f (0x9559 bytes) 000446 1713425433.218980 MIRROR_EXTEND to 4 mirrors 000448 1713425433.327824 write 0x02dd42 thru 0x0373d1 (0x9690 bytes) 000449 1713425433.336772 fallocate 0x020779 thru 0x02640e (0x5c96 bytes) 000450 1713425433.338065 punch from 0x004aaa to 0x00f720 (0xac77 bytes) 000451 1713425433.339400 write 0x01ac90 thru 0x0200eb (0x545c bytes) 000452 1713425433.342493 write 0x02da5b thru 0x034f07 (0x74ad bytes) 000453 1713425433.344205 MIRROR_EXTEND to 5 mirrors 000454 1713425433.449086 read 0x02e626 thru 0x0373d1 (0x8dac bytes) 000455 1713425433.452856 punch from 0x022581 to 0x022e5a (0x8da bytes) 000457 1713425433.460531 mapwrite 0x02e795 thru 0x03c96e (0xe1da bytes) 000458 1713425433.482202 punch from 0x00b419 to 0x01a8fd (0xf4e5 bytes) 000459 1713425433.483720 mapread 0x000be6 thru 0x00a320 (0x973b bytes) 000460 1713425433.488338 mapwrite 0x01e815 thru 0x01f728 (0xf14 bytes) 000461 1713425433.493451 write 0x016827 thru 0x01ae34 (0x460e bytes) 000462 1713425433.495168 write 0x01b6ff thru 0x01dd7d (0x267f bytes) 000463 1713425433.495713 read 0x009eee thru 0x013dd1 (0x9ee4 bytes) 000464 1713425433.496036 punch from 0x03a12b to 0x03c96e (0x2844 bytes) 000465 1713425433.497505 write 0x0342b4 thru 0x03ffff (0xbd4c bytes) 000466 1713425433.498339 mapread 0x01dc70 thru 0x02b75d (0xdaee bytes) 000467 1713425433.500239 trunc from 0x040000 to 0x0115ad (0x2ea52 bytes) 000468 1713425433.504136 read 0x00cd2e thru 0x00ce85 (0x158 bytes) 000469 1713425433.504439 mapread 0x010419 thru 0x0115ad (0x1195 bytes) 000470 1713425433.505006 read 0x00133c thru 0x00e2b2 (0xcf77 bytes) 000471 1713425433.505325 write 0x0176b0 thru 0x022742 (0xb093 bytes) 000472 1713425433.506050 fallocate 0x007e45 thru 0x00c8fd (0x4ab9 bytes) 000473 1713425433.507586 write 0x02e182 thru 0x03735d (0x91dc bytes) 000475 1713425433.509873 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000476 1713425433.555294 read 0x033c4b thru 0x0359cb (0x1d81 bytes) 000477 1713425433.556905 mapwrite 0x0265ae thru 0x02a4f3 (0x3f46 bytes) 000478 1713425433.566829 mapread 0x030e19 thru 0x03735d (0x6545 bytes) 000480 1713425433.569840 trunc from 0x03735e to 0x02027c (0x170e1 bytes) 000481 1713425433.575256 fallocate 0x03c6f6 thru 0x03ffff (0x390a bytes) 000482 1713425433.576936 mapread 0x00c2b6 thru 0x0106a7 (0x43f2 bytes) 000483 1713425433.580328 fallocate 0x01b172 thru 0x024144 (0x8fd3 bytes) 000485 1713425433.582526 trunc from 0x024145 to 0x017d49 (0xc3fb bytes) 000486 1713425433.588082 MIRROR_RESYNC 4 mirrors 000487 1713425433.703772 trunc from 0x017d4a to 0x037947 (0x1fbfe bytes) 000488 1713425433.718532 MIRROR_EXTEND to 5 mirrors 000489 1713425433.828694 fallocate 0x00dbfa thru 0x011971 (0x3d78 bytes) 000490 1713425433.837425 mapwrite 0x0176f7 thru 0x0182d1 (0xbdb bytes) 000492 1713425433.845330 MIRROR_EXTEND to 6 mirrors 000493 1713425433.952910 read 0x003901 thru 0x012a2b (0xf12b bytes) 000495 1713425433.958220 mapread 0x003754 thru 0x006a8c (0x3339 bytes) 000498 1713425433.965094 read 0x02702d thru 0x02e528 (0x74fc bytes) 000499 1713425433.969226 read 0x01bcfd thru 0x0299f2 (0xdcf6 bytes) 000500 1713425433.971447 write 0x0331f9 thru 0x03aeb1 (0x7cb9 bytes) 000501 1713425433.982895 fallocate 0x00bd9a thru 0x0150b5 (0x931c bytes) 000502 1713425433.984604 mapread 0x022422 thru 0x02c994 (0xa573 bytes) 000503 1713425433.985117 write 0x0102b7 thru 0x01e8bc (0xe606 bytes) 000504 1713425433.986048 punch from 0x02982b to 0x036201 (0xc9d7 bytes) 000505 1713425433.990206 mapread 0x00a304 thru 0x00e2d2 (0x3fcf bytes) 000506 1713425433.990635 read 0x027c83 thru 0x0331b7 (0xb535 bytes) 000507 1713425433.992844 fallocate 0x01e7a1 thru 0x0225f6 (0x3e56 bytes) 000508 1713425433.994216 punch from 0x038909 to 0x03aeb1 (0x25a9 bytes) 000509 1713425433.995457 write 0x005165 thru 0x009969 (0x4805 bytes) 000510 1713425433.996086 write 0x034a6a thru 0x03ffff (0xb596 bytes) 000514 1713425433.998692 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000515 1713425434.044525 trunc from 0x040000 to 0x039fa1 (0x605e bytes) 000516 1713425434.054621 write 0x03c35d thru 0x03ffff (0x3ca3 bytes) 000517 1713425434.056368 MIRROR_EXTEND to 6 mirrors 000518 1713425434.194606 mapwrite 0x01b97c thru 0x0204d7 (0x4b5c bytes) 000519 1713425434.230527 MIRROR_RESYNC 6 mirrors 000520 1713425434.468667 read 0x0058b3 thru 0x0128b1 (0xcfff bytes) 000522 1713425434.470822 punch from 0x0194b5 to 0x01d1ea (0x3d36 bytes) 000523 1713425434.478881 fallocate 0x0105f6 thru 0x01dd21 (0xd72c bytes) 000524 1713425434.480465 mapread 0x006cc4 thru 0x00df1d (0x725a bytes) 000525 1713425434.485165 trunc from 0x040000 to 0x0081f3 (0x37e0c bytes) 000526 1713425434.491499 read 0x00741a thru 0x0081f3 (0xdda bytes) 000527 1713425434.491958 fallocate 0x00b8e4 thru 0x01afbd (0xf6da bytes) 000529 1713425434.493849 mapwrite 0x035005 thru 0x037d4b (0x2d47 bytes) 000530 1713425434.506636 trunc from 0x037d4c to 0x0212c6 (0x16a85 bytes) 000531 1713425434.510328 write 0x006e5a thru 0x009bfe (0x2da5 bytes) 000532 1713425434.510889 mapwrite 0x006154 thru 0x015bad (0xfa5a bytes) 000533 1713425434.518813 read 0x020ac4 thru 0x0212c6 (0x803 bytes) 000534 1713425434.519118 mapwrite 0x01f6b7 thru 0x021c96 (0x25e0 bytes) 000535 1713425434.526936 read 0x01dd9e thru 0x021c96 (0x3ef9 bytes) 000537 1713425434.527552 trunc from 0x021c97 to 0x02f941 (0xdcab bytes) 000540 1713425434.532720 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000542 1713425434.580009 fallocate 0x011a26 thru 0x01814e (0x6729 bytes) 000544 1713425434.582482 mapread 0x008c51 thru 0x00abfd (0x1fad bytes) 000545 1713425434.587800 mapwrite 0x032cf6 thru 0x03ffff (0xd30a bytes) 000546 1713425434.603445 punch from 0x007372 to 0x012ca8 (0xb937 bytes) 000547 1713425434.605245 fallocate 0x03d7dd thru 0x03ffff (0x2823 bytes) 000548 1713425434.606732 mapread 0x0088ff thru 0x016c5d (0xe35f bytes) 000549 1713425434.611504 trunc from 0x040000 to 0x01f447 (0x20bb8 bytes) 000550 1713425434.615611 trunc from 0x01f448 to 0x01c10b (0x333c bytes) 000551 1713425434.619852 MIRROR_RESYNC 5 mirrors 000552 1713425434.816831 punch from 0x01bc69 to 0x01c10b (0x4a3 bytes) 000553 1713425434.826260 mapwrite 0x01d790 thru 0x02916f (0xb9e0 bytes) 000555 1713425434.849806 MIRROR_EXTEND to 6 mirrors 000556 1713425434.969070 write 0x03972f thru 0x03ffff (0x68d1 bytes) 000557 1713425434.978464 mapwrite 0x02e734 thru 0x034b28 (0x63f5 bytes) 000558 1713425434.989423 mapread 0x0236dc thru 0x029aff (0x6424 bytes) 000560 1713425434.992445 mapwrite 0x009f37 thru 0x011e3e (0x7f08 bytes) 000562 1713425435.000018 trunc from 0x040000 to 0x029967 (0x16698 bytes) 000563 1713425435.004956 trunc from 0x029968 to 0x0134ff (0x16468 bytes) 000565 1713425435.009639 trunc from 0x013500 to 0x00dc42 (0x58bd bytes) 000566 1713425435.014382 fallocate 0x03d2c8 thru 0x03ffff (0x2d38 bytes) 000567 1713425435.016289 punch from 0x027810 to 0x02988a (0x207b bytes) 000568 1713425435.018895 MIRROR_RESYNC 6 mirrors 000569 1713425435.284206 write 0x03bad6 thru 0x03ffff (0x452a bytes) 000571 1713425435.293735 punch from 0x01a606 to 0x0243dd (0x9dd8 bytes) 000573 1713425435.295834 mapwrite 0x018298 thru 0x022265 (0x9fce bytes) 000574 1713425435.308197 fallocate 0x0211b4 thru 0x0257b8 (0x4605 bytes) 000575 1713425435.309886 mapread 0x00d21d thru 0x01637e (0x9162 bytes) 000576 1713425435.313152 MIRROR_RESYNC 6 mirrors 000577 1713425435.579533 mapwrite 0x031bcd thru 0x03db09 (0xbf3d bytes) 000578 1713425435.612521 read 0x02c5c4 thru 0x036b36 (0xa573 bytes) 000579 1713425435.612878 trunc from 0x040000 to 0x0210dd (0x1ef22 bytes) 000580 1713425435.622610 fallocate 0x00e0a9 thru 0x010d07 (0x2c5f bytes) 000583 1713425435.625113 write 0x03b947 thru 0x03ffff (0x46b9 bytes) 000584 1713425435.626165 write 0x008628 thru 0x00cef9 (0x48d2 bytes) 000585 1713425435.630430 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000586 1713425435.681790 read 0x03e4ea thru 0x03ffff (0x1b16 bytes) 000587 1713425435.683651 punch from 0x03b712 to 0x03ffff (0x48ee bytes) 000588 1713425435.685512 fallocate 0x000593 thru 0x00926b (0x8cd9 bytes) 000589 1713425435.688946 MIRROR_EXTEND to 6 mirrors 000590 1713425435.813388 trunc from 0x040000 to 0x028033 (0x17fcc bytes) 000591 1713425435.828794 MIRROR_RESYNC 6 mirrors 000592 1713425436.098636 trunc from 0x028034 to 0x025504 (0x2b2f bytes) 000593 1713425436.108912 write 0x000a7d thru 0x004830 (0x3db4 bytes) 000594 1713425436.111906 write 0x020765 thru 0x0238c4 (0x3160 bytes) 000596 1713425436.116388 fallocate 0x02265d thru 0x032523 (0xfec7 bytes) 000597 1713425436.118334 read 0x0210c7 thru 0x025504 (0x443e bytes) 000598 1713425436.120343 fallocate 0x0073a6 thru 0x011fc6 (0xac21 bytes) 000599 1713425436.122221 mapwrite 0x0048f1 thru 0x007a89 (0x3199 bytes) 000600 1713425436.129785 punch from 0x01c430 to 0x020c56 (0x4827 bytes) 000601 1713425436.131648 trunc from 0x025505 to 0x01e8ba (0x6c4a bytes) 000602 1713425436.137549 write 0x03d14e thru 0x03ffff (0x2eb2 bytes) 000603 1713425436.138191 write 0x02c2a7 thru 0x02ed36 (0x2a90 bytes) 000605 1713425436.140941 write 0x012249 thru 0x01e3eb (0xc1a3 bytes) 000606 1713425436.147046 MIRROR_EXTEND to 7 mirrors 000607 1713425436.283131 trunc from 0x040000 to 0x004dfb (0x3b204 bytes) 000610 1713425436.297106 fallocate 0x032860 thru 0x03e299 (0xba3a bytes) 000611 1713425436.298902 read 0x00853f thru 0x0153be (0xce80 bytes) 000612 1713425436.301517 read 0x01557d thru 0x020ca0 (0xb724 bytes) 000613 1713425436.304486 read 0x0222f5 thru 0x02cbff (0xa90b bytes) 000614 1713425436.305815 MIRROR_EXTEND to 8 mirrors 000615 1713425436.417155 MIRROR_RESYNC 8 mirrors 000618 1713425436.589842 MIRROR_EXTEND to 9 mirrors 000619 1713425436.716308 mapread 0x0206d1 thru 0x029fef (0x991f bytes) 000620 1713425436.721412 mapwrite 0x03a6f0 thru 0x03c53d (0x1e4e bytes) 000622 1713425436.751814 fallocate 0x0047ee thru 0x0066cb (0x1ede bytes) 000623 1713425436.754524 MIRROR_EXTEND to 10 mirrors 000625 1713425436.865095 mapread 0x027c71 thru 0x0355a2 (0xd932 bytes) 000626 1713425436.877682 mapwrite 0x01ec86 thru 0x02d0c6 (0xe441 bytes) 000627 1713425436.919694 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000628 1713425436.989682 read 0x0330d5 thru 0x038f95 (0x5ec1 bytes) 000629 1713425436.993009 trunc from 0x03e29a to 0x021d34 (0x1c565 bytes) 000630 1713425437.007935 read 0x0201e1 thru 0x021d34 (0x1b54 bytes) 000631 1713425437.012309 read 0x014ea3 thru 0x01ed0a (0x9e68 bytes) 000632 1713425437.014704 MIRROR_EXTEND to 10 mirrors 000633 1713425437.150777 write 0x02fac8 thru 0x0342ca (0x4803 bytes) 000634 1713425437.161491 mapwrite 0x029e01 thru 0x02bc24 (0x1e24 bytes) 000635 1713425437.172404 fallocate 0x01464d thru 0x01c99f (0x8353 bytes) 000636 1713425437.174372 trunc from 0x0342cb to 0x0326da (0x1bf0 bytes) 000637 1713425437.183452 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000638 1713425437.245218 mapread 0x002869 thru 0x010ab6 (0xe24e bytes) 000640 1713425437.256154 mapread 0x016800 thru 0x025539 (0xed3a bytes) 000641 1713425437.262545 fallocate 0x019856 thru 0x02597d (0xc128 bytes) 000642 1713425437.265076 read 0x02ce08 thru 0x0326da (0x58d3 bytes) 000643 1713425437.268582 trunc from 0x0326db to 0x021836 (0x10ea4 bytes) 000644 1713425437.276660 fallocate 0x023f3f thru 0x02fcd5 (0xbd97 bytes) 000645 1713425437.278614 mapread 0x01af65 thru 0x021836 (0x68d2 bytes) 000646 1713425437.279267 fallocate 0x003543 thru 0x00bf2a (0x89e8 bytes) 000647 1713425437.280866 mapread 0x019833 thru 0x021836 (0x8004 bytes) 000648 1713425437.281424 write 0x025bd1 thru 0x02bbe4 (0x6014 bytes) 000649 1713425437.282300 mapread 0x024790 thru 0x02bbe4 (0x7455 bytes) 000650 1713425437.284240 fallocate 0x022306 thru 0x0265ad (0x42a8 bytes) 000651 1713425437.286111 mapwrite 0x017cd1 thru 0x02474f (0xca7f bytes) 000652 1713425437.294750 mapwrite 0x02bd9a thru 0x03b915 (0xfb7c bytes) 000653 1713425437.315977 mapwrite 0x004b2a thru 0x00717d (0x2654 bytes) 000654 1713425437.323242 MIRROR_EXTEND to 10 mirrors 000656 1713425437.478332 read 0x006125 thru 0x015ccb (0xfba7 bytes) 000657 1713425437.484326 MIRROR_EXTEND to 11 mirrors 000658 1713425437.647052 mapwrite 0x01708f thru 0x023b5c (0xcace bytes) 000659 1713425437.686193 read 0x008776 thru 0x00c882 (0x410d bytes) 000660 1713425437.688618 mapread 0x021c3f thru 0x025644 (0x3a06 bytes) 000661 1713425437.689100 write 0x02c220 thru 0x02d131 (0xf12 bytes) 000662 1713425437.692818 punch from 0x00c715 to 0x019977 (0xd263 bytes) 000664 1713425437.694898 punch from 0x01aff2 to 0x022a25 (0x7a34 bytes) 000665 1713425437.696841 mapread 0x00365f thru 0x006b55 (0x34f7 bytes) 000666 1713425437.699584 read 0x02d19b thru 0x035e80 (0x8ce6 bytes) 000667 1713425437.702790 write 0x025d63 thru 0x034efc (0xf19a bytes) 000668 1713425437.703984 mapwrite 0x02f094 thru 0x036cb4 (0x7c21 bytes) 000669 1713425437.709456 trunc from 0x03b916 to 0x01b967 (0x1ffae bytes) 000670 1713425437.718106 punch from 0x015825 to 0x01b094 (0x5870 bytes) 000673 1713425437.720158 read 0x0115d2 thru 0x017704 (0x6133 bytes) 000674 1713425437.722736 trunc from 0x01b968 to 0x0018d4 (0x1a093 bytes) 000675 1713425437.726971 punch from 0x0015d7 to 0x0018d4 (0x2fe bytes) 000676 1713425437.728397 fallocate 0x010aca thru 0x01e9bd (0xdef4 bytes) 000677 1713425437.730499 trunc from 0x01e9be to 0x03f4da (0x20b1d bytes) 000679 1713425437.734661 trunc from 0x03f4db to 0x032bbf (0xc91b bytes) 000682 1713425437.738790 punch from 0x00d314 to 0x016283 (0x8f70 bytes) 000683 1713425437.740313 write 0x010824 thru 0x01478e (0x3f6b bytes) 000684 1713425437.744158 read 0x00456c thru 0x00a890 (0x6325 bytes) 000685 1713425437.746640 read 0x02788a thru 0x032bbf (0xb336 bytes) 000686 1713425437.747129 punch from 0x006fc5 to 0x011bfc (0xac38 bytes) 000687 1713425437.750725 punch from 0x000467 to 0x00549e (0x5038 bytes) 000688 1713425437.753599 MIRROR_RESYNC 11 mirrors 000689 1713425438.154843 MIRROR_EXTEND to 12 mirrors 000693 1713425438.298824 write 0x01f825 thru 0x01ffdd (0x7b9 bytes) 000694 1713425438.310043 mapwrite 0x01697f thru 0x019194 (0x2816 bytes) 000695 1713425438.317950 write 0x020756 thru 0x02d093 (0xc93e bytes) 000697 1713425438.320266 write 0x009ad1 thru 0x00d95c (0x3e8c bytes) 000699 1713425438.322938 punch from 0x01efd8 to 0x0219e8 (0x2a11 bytes) 000700 1713425438.326570 fallocate 0x013ea9 thru 0x01f0c9 (0xb221 bytes) 000701 1713425438.329393 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000702 1713425438.376077 fallocate 0x03019e thru 0x0302d3 (0x136 bytes) 000703 1713425438.377946 trunc from 0x032bc0 to 0x036a2f (0x3e70 bytes) 000704 1713425438.384527 MIRROR_EXTEND to 12 mirrors 000705 1713425438.560043 write 0x03277b thru 0x03c6e9 (0x9f6f bytes) 000707 1713425438.573459 trunc from 0x03c6ea to 0x03869c (0x404d bytes) 000708 1713425438.579913 trunc from 0x03869d to 0x00fdd6 (0x288c6 bytes) 000709 1713425438.584475 write 0x003f3d thru 0x00e1b4 (0xa278 bytes) 000710 1713425438.588144 mapread 0x00ad09 thru 0x00fdd6 (0x50ce bytes) 000712 1713425438.590762 write 0x01d5d0 thru 0x02c90d (0xf33e bytes) 000713 1713425438.591910 punch from 0x02bb29 to 0x02c90d (0xde5 bytes) 000714 1713425438.596734 write 0x022d66 thru 0x027692 (0x492d bytes) 000715 1713425438.597446 mapwrite 0x039fb6 thru 0x03cff0 (0x303b bytes) 000717 1713425438.612935 mapwrite 0x02e0e5 thru 0x032e97 (0x4db3 bytes) 000718 1713425438.620495 mapwrite 0x038524 thru 0x03d96c (0x5449 bytes) 000719 1713425438.638161 read 0x002126 thru 0x010952 (0xe82d bytes) 000720 1713425438.641987 write 0x039a10 thru 0x03ffff (0x65f0 bytes) 000721 1713425438.642868 punch from 0x006e6b to 0x00ac91 (0x3e27 bytes) 000722 1713425438.644462 trunc from 0x040000 to 0x00a0b5 (0x35f4a bytes) 000723 1713425438.649683 read 0x007c8a thru 0x00a0b5 (0x242c bytes) 000724 1713425438.653302 read 0x001674 thru 0x00a0b5 (0x8a42 bytes) 000725 1713425438.653979 mapwrite 0x00e4af thru 0x0154cb (0x701d bytes) 000726 1713425438.672784 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000727 1713425438.757521 punch from 0x000f02 to 0x00619f (0x529e bytes) 000728 1713425438.760746 read 0x00cc46 thru 0x0154cb (0x8886 bytes) 000729 1713425438.765237 write 0x01eece thru 0x0261a1 (0x72d4 bytes) 000730 1713425438.768790 trunc from 0x0261a2 to 0x004f6e (0x21233 bytes) 000731 1713425438.776478 read 0x00214c thru 0x004f6e (0x2e23 bytes) 000733 1713425438.781069 mapwrite 0x007790 thru 0x00798f (0x200 bytes) 000734 1713425438.799597 mapwrite 0x02044f thru 0x0267e5 (0x6397 bytes) 000735 1713425438.820568 trunc from 0x0267e6 to 0x033b54 (0xd36f bytes) 000736 1713425438.827017 mapwrite 0x026106 thru 0x0351e5 (0xf0e0 bytes) 000737 1713425438.845414 read 0x0053ef thru 0x011b8c (0xc79e bytes) 000738 1713425438.846059 read 0x018593 thru 0x01da90 (0x54fe bytes) 000739 1713425438.846441 mapwrite 0x01dc2a thru 0x022bce (0x4fa5 bytes) 000740 1713425438.853721 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000741 1713425438.912677 mapwrite 0x008f8e thru 0x014cf5 (0xbd68 bytes) 000743 1713425438.927973 mapread 0x02bf8d thru 0x0351e5 (0x9259 bytes) 000744 1713425438.933095 mapread 0x01b92f thru 0x0274db (0xbbad bytes) 000747 1713425438.938094 write 0x0228e5 thru 0x02bbe3 (0x92ff bytes) 000748 1713425438.940248 write 0x0222bf thru 0x029803 (0x7545 bytes) 000749 1713425438.940876 punch from 0x01def6 to 0x028a76 (0xab81 bytes) 000751 1713425438.946456 MIRROR_EXTEND to 11 mirrors 000752 1713425439.070084 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000753 1713425439.128537 read 0x02f4df thru 0x031a72 (0x2594 bytes) 000754 1713425439.133369 mapread 0x00f37e thru 0x01f09e (0xfd21 bytes) 000755 1713425439.144161 punch from 0x02e28c to 0x031143 (0x2eb8 bytes) 000756 1713425439.153606 mapread 0x000eac thru 0x00a328 (0x947d bytes) 000757 1713425439.157925 punch from 0x023ad3 to 0x02d521 (0x9a4f bytes) 000760 1713425439.160073 fallocate 0x038403 thru 0x03bf4d (0x3b4b bytes) 000761 1713425439.161795 trunc from 0x03bf4e to 0x00ac5d (0x312f0 bytes) 000762 1713425439.168169 MIRROR_EXTEND to 11 mirrors 000764 1713425439.269381 mapread 0x006b40 thru 0x009aad (0x2f6e bytes) 000765 1713425439.276744 punch from 0x000bf7 to 0x004a26 (0x3e30 bytes) 000767 1713425439.286940 MIRROR_RESYNC 11 mirrors 000768 1713425439.563426 trunc from 0x00ac5e to 0x02a5b0 (0x1f953 bytes) 000769 1713425439.576483 mapwrite 0x022ae6 thru 0x031436 (0xe951 bytes) 000770 1713425439.595304 punch from 0x00432a to 0x011293 (0xcf6a bytes) 000771 1713425439.596690 fallocate 0x031440 thru 0x03a056 (0x8c17 bytes) 000772 1713425439.597903 fallocate 0x019997 thru 0x01b733 (0x1d9d bytes) 000773 1713425439.599153 mapread 0x02fdc6 thru 0x031436 (0x1671 bytes) 000775 1713425439.599948 read 0x00911f thru 0x012a9f (0x9981 bytes) 000777 1713425439.604009 mapwrite 0x0346d7 thru 0x03b80a (0x7134 bytes) 000778 1713425439.618014 write 0x01344b thru 0x016312 (0x2ec8 bytes) 000779 1713425439.621064 read 0x007ce3 thru 0x015d89 (0xe0a7 bytes) 000780 1713425439.625455 MIRROR_RESYNC 11 mirrors 000781 1713425439.949374 fallocate 0x0083fd thru 0x015907 (0xd50b bytes) 000782 1713425439.957642 mapwrite 0x03bb16 thru 0x03ffff (0x44ea bytes) 000784 1713425439.973887 trunc from 0x040000 to 0x03fe1b (0x1e4 bytes) 000786 1713425439.978783 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000787 1713425440.022549 write 0x00bcf2 thru 0x0134d2 (0x77e1 bytes) 000789 1713425440.026595 trunc from 0x03fe1c to 0x00336f (0x3caac bytes) 000790 1713425440.030309 trunc from 0x003370 to 0x03950c (0x3619d bytes) 000792 1713425440.033672 fallocate 0x035091 thru 0x03dc39 (0x8ba9 bytes) 000794 1713425440.035332 mapread 0x00d95e thru 0x019810 (0xbeb3 bytes) 000795 1713425440.039876 mapwrite 0x0370dc thru 0x03ffff (0x8f24 bytes) 000796 1713425440.053012 fallocate 0x034500 thru 0x03a80b (0x630c bytes) 000797 1713425440.054506 fallocate 0x021f1a thru 0x03147c (0xf563 bytes) 000798 1713425440.055912 mapwrite 0x023391 thru 0x029189 (0x5df9 bytes) 000799 1713425440.063150 mapwrite 0x01f1d8 thru 0x02d7ac (0xe5d5 bytes) 000800 1713425440.070230 punch from 0x03d835 to 0x03ffff (0x27cb bytes) 000801 1713425440.071649 fallocate 0x034a5e thru 0x03b8e6 (0x6e89 bytes) 000802 1713425440.073071 read 0x01ae34 thru 0x01c6a9 (0x1876 bytes) 000803 1713425440.073453 read 0x03d16f thru 0x03ffff (0x2e91 bytes) 000804 1713425440.075766 read 0x00d4c3 thru 0x00db17 (0x655 bytes) 000805 1713425440.077341 MIRROR_EXTEND to 11 mirrors 000806 1713425440.198224 punch from 0x015e23 to 0x020cca (0xaea8 bytes) 000807 1713425440.207795 punch from 0x00687b to 0x00daea (0x7270 bytes) 000808 1713425440.210571 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000809 1713425440.253006 mapread 0x014395 thru 0x023f74 (0xfbe0 bytes) 000810 1713425440.259244 mapread 0x0350a5 thru 0x03fc17 (0xab73 bytes) 000812 1713425440.263560 mapwrite 0x00bd73 thru 0x00ce88 (0x1116 bytes) 000813 1713425440.271989 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000814 1713425440.318450 trunc from 0x040000 to 0x02d15c (0x12ea3 bytes) 000815 1713425440.326136 mapread 0x010478 thru 0x01625f (0x5de8 bytes) 000816 1713425440.330765 trunc from 0x02d15d to 0x01210c (0x1b050 bytes) 000817 1713425440.334658 read 0x002d98 thru 0x00f5d0 (0xc839 bytes) 000818 1713425440.337993 MIRROR_RESYNC 9 mirrors 000820 1713425440.569433 trunc from 0x01210d to 0x017dc1 (0x5cb5 bytes) 000821 1713425440.581803 trunc from 0x017dc2 to 0x004c5d (0x13164 bytes) 000822 1713425440.586662 fallocate 0x01400e thru 0x01e082 (0xa075 bytes) 000823 1713425440.588053 mapread 0x0004f6 thru 0x004c5d (0x4768 bytes) 000824 1713425440.593739 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000825 1713425440.646397 mapread 0x002d3e thru 0x004c5d (0x1f20 bytes) 000826 1713425440.651256 MIRROR_EXTEND to 9 mirrors 000827 1713425440.744068 trunc from 0x004c5e to 0x034a8c (0x2fe2f bytes) 000828 1713425440.757659 mapread 0x00ec87 thru 0x0125a7 (0x3921 bytes) 000829 1713425440.762328 write 0x014bbb thru 0x01bdda (0x7220 bytes) 000830 1713425440.764114 mapread 0x00efb3 thru 0x017db9 (0x8e07 bytes) 000831 1713425440.764568 write 0x031fe1 thru 0x03c020 (0xa040 bytes) 000833 1713425440.767039 read 0x01c5db thru 0x02bd60 (0xf786 bytes) 000834 1713425440.770494 MIRROR_EXTEND to 10 mirrors 000835 1713425440.872980 mapread 0x00758b thru 0x00f1c9 (0x7c3f bytes) 000836 1713425440.880202 trunc from 0x03c021 to 0x0345dc (0x7a44 bytes) 000837 1713425440.896326 MIRROR_RESYNC 10 mirrors 000838 1713425441.197892 trunc from 0x0345dd to 0x00377d (0x30e5f bytes) 000839 1713425441.207247 mapwrite 0x0340dd thru 0x035db0 (0x1cd4 bytes) 000840 1713425441.218702 mapread 0x01d9fa thru 0x0229b7 (0x4fbe bytes) 000841 1713425441.221411 fallocate 0x016d11 thru 0x01f3cc (0x86bc bytes) 000842 1713425441.223990 MIRROR_RESYNC 10 mirrors 000843 1713425441.458154 mapread 0x01400d thru 0x01b478 (0x746c bytes) 000845 1713425441.463877 mapwrite 0x00a0db thru 0x0143be (0xa2e4 bytes) 000846 1713425441.500747 mapwrite 0x034bcc thru 0x03ccaa (0x80df bytes) 000847 1713425441.520723 mapread 0x00df18 thru 0x01cac4 (0xebad bytes) 000849 1713425441.524494 write 0x008832 thru 0x00bb75 (0x3344 bytes) 000850 1713425441.525295 trunc from 0x03ccab to 0x023b5e (0x1914c bytes) 000851 1713425441.530629 mapwrite 0x035149 thru 0x03ffff (0xaeb7 bytes) 000852 1713425441.547672 trunc from 0x040000 to 0x01193c (0x2e6c3 bytes) 000853 1713425441.552758 punch from 0x00b282 to 0x01193c (0x66bb bytes) 000856 1713425441.554843 write 0x019793 thru 0x01a4ce (0xd3c bytes) 000858 1713425441.557224 MIRROR_RESYNC 10 mirrors 000859 1713425441.935494 mapread 0x009a44 thru 0x015f84 (0xc541 bytes) 000860 1713425441.941676 read 0x01843c thru 0x01a4ce (0x2093 bytes) 000863 1713425441.942157 write 0x0302fb thru 0x03b53f (0xb245 bytes) 000864 1713425441.950307 mapwrite 0x018bbe thru 0x01952a (0x96d bytes) 000867 1713425441.958521 read 0x00f412 thru 0x017bc9 (0x87b8 bytes) 000868 1713425441.960859 mapread 0x02104b thru 0x0236c0 (0x2676 bytes) 000869 1713425441.961295 mapread 0x025366 thru 0x034da3 (0xfa3e bytes) 000871 1713425441.961891 fallocate 0x01a557 thru 0x027bd2 (0xd67c bytes) 000872 1713425441.963484 mapwrite 0x03c43f thru 0x03dfa5 (0x1b67 bytes) 000873 1713425441.977678 mapwrite 0x01fd73 thru 0x029269 (0x94f7 bytes) 000874 1713425441.982687 read 0x03b299 thru 0x03dfa5 (0x2d0d bytes) 000877 1713425441.983442 punch from 0x0093ac to 0x00ef69 (0x5bbe bytes) 000878 1713425441.984807 trunc from 0x03dfa6 to 0x00e136 (0x2fe6f bytes) 000879 1713425441.988552 trunc from 0x00e137 to 0x01bdc1 (0xdc8b bytes) 000880 1713425441.992104 mapread 0x015026 thru 0x015780 (0x75b bytes) 000881 1713425441.994508 fallocate 0x020795 thru 0x02b113 (0xa97f bytes) 000883 1713425441.997305 MIRROR_RESYNC 10 mirrors 000884 1713425442.253031 read 0x00c6c9 thru 0x016377 (0x9caf bytes) 000885 1713425442.255079 mapwrite 0x028041 thru 0x033133 (0xb0f3 bytes) 000886 1713425442.274222 write 0x026ca7 thru 0x031617 (0xa971 bytes) 000887 1713425442.274939 punch from 0x017318 to 0x01ff44 (0x8c2d bytes) 000888 1713425442.276558 write 0x03a215 thru 0x03bd6c (0x1b58 bytes) 000889 1713425442.277054 write 0x01b755 thru 0x01e9ee (0x329a bytes) 000890 1713425442.279400 read 0x01daec thru 0x02c82c (0xed41 bytes) 000891 1713425442.279785 mapread 0x0273d8 thru 0x028cd3 (0x18fc bytes) 000892 1713425442.280155 read 0x016171 thru 0x0182c8 (0x2158 bytes) 000893 1713425442.280577 write 0x003830 thru 0x011501 (0xdcd2 bytes) 000894 1713425442.283515 mapread 0x00d356 thru 0x01964b (0xc2f6 bytes) 000895 1713425442.283985 mapread 0x01a058 thru 0x02402b (0x9fd4 bytes) 000896 1713425442.284527 write 0x0212b5 thru 0x030827 (0xf573 bytes) 000897 1713425442.285452 read 0x027891 thru 0x02b7e8 (0x3f58 bytes) 000898 1713425442.285794 write 0x008613 thru 0x00b1f7 (0x2be5 bytes) 000900 1713425442.286686 trunc from 0x03bd6d to 0x016c22 (0x2514a bytes) 000902 1713425442.292045 write 0x003a50 thru 0x011646 (0xdbf7 bytes) 000903 1713425442.292816 read 0x012ccd thru 0x01483a (0x1b6e bytes) 000904 1713425442.293245 read 0x00e968 thru 0x016c22 (0x82bb bytes) 000906 1713425442.293714 mapread 0x00d701 thru 0x012c10 (0x5510 bytes) 000907 1713425442.294188 read 0x010719 thru 0x016c22 (0x650a bytes) 000909 1713425442.294754 fallocate 0x00e471 thru 0x0141f9 (0x5d89 bytes) 000910 1713425442.296452 fallocate 0x002345 thru 0x00d8fd (0xb5b9 bytes) 000912 1713425442.297955 read 0x008606 thru 0x00ddfe (0x57f9 bytes) 000915 1713425442.298648 mapread 0x00a6e1 thru 0x01427d (0x9b9d bytes) 000916 1713425442.299131 fallocate 0x025467 thru 0x027a29 (0x25c3 bytes) 000917 1713425442.300655 write 0x02bb13 thru 0x0342d2 (0x87c0 bytes) 000918 1713425442.301383 mapwrite 0x02c6de thru 0x03288c (0x61af bytes) 000919 1713425442.308167 mapwrite 0x006f97 thru 0x01594a (0xe9b4 bytes) 000920 1713425442.315980 MIRROR_EXTEND to 11 mirrors 000921 1713425442.424626 fallocate 0x037c09 thru 0x03dc2d (0x6025 bytes) 000924 1713425442.434746 write 0x005407 thru 0x00b71f (0x6319 bytes) 000926 1713425442.438311 trunc from 0x03dc2e to 0x035e10 (0x7e1d bytes) 000927 1713425442.442830 fallocate 0x027dae thru 0x036b44 (0xed97 bytes) 000928 1713425442.444492 fallocate 0x02251a thru 0x025149 (0x2c30 bytes) 000929 1713425442.446035 read 0x014043 thru 0x022413 (0xe3d1 bytes) 000930 1713425442.448346 write 0x004ad8 thru 0x008f48 (0x4471 bytes) 000931 1713425442.450370 fallocate 0x001c58 thru 0x0061b8 (0x4561 bytes) 000932 1713425442.451860 punch from 0x007d42 to 0x01317d (0xb43c bytes) 000933 1713425442.455070 mapread 0x03398b thru 0x036b44 (0x31ba bytes) 000934 1713425442.457312 fallocate 0x036b1f thru 0x03ffff (0x94e1 bytes) 000935 1713425442.458558 mapread 0x02fad3 thru 0x034f56 (0x5484 bytes) 000936 1713425442.459137 fallocate 0x020209 thru 0x024972 (0x476a bytes) 000937 1713425442.460442 punch from 0x03a251 to 0x03ffff (0x5daf bytes) 000938 1713425442.461752 fallocate 0x03ccb4 thru 0x03ffff (0x334c bytes) 000940 1713425442.464687 MIRROR_EXTEND to 12 mirrors 000941 1713425442.582949 write 0x0151e0 thru 0x01c914 (0x7735 bytes) 000942 1713425442.593953 write 0x0045ff thru 0x006de4 (0x27e6 bytes) 000943 1713425442.596356 fallocate 0x03ee3f thru 0x03ffff (0x11c1 bytes) 000944 1713425442.597639 write 0x03b0ea thru 0x03ffff (0x4f16 bytes) 000945 1713425442.599941 trunc from 0x040000 to 0x0056ec (0x3a913 bytes) 000946 1713425442.608416 mapwrite 0x004763 thru 0x0047c1 (0x5f bytes) 000947 1713425442.612827 read 0x00032a thru 0x0056ec (0x53c3 bytes) 000948 1713425442.614368 fallocate 0x0284bc thru 0x0376dd (0xf222 bytes) 000949 1713425442.616019 mapwrite 0x02dbc8 thru 0x03b185 (0xd5be bytes) 000950 1713425442.629856 read 0x00df1d thru 0x0190f2 (0xb1d6 bytes) 000952 1713425442.632900 read 0x00216c thru 0x00bda4 (0x9c39 bytes) 000953 1713425442.633231 read 0x03616d thru 0x03b185 (0x5019 bytes) 000954 1713425442.633527 punch from 0x01e653 to 0x01e977 (0x325 bytes) 000956 1713425442.635003 trunc from 0x03b186 to 0x01ce3c (0x1e349 bytes) 000957 1713425442.638604 mapread 0x01c1d8 thru 0x01ce3c (0xc65 bytes) 000958 1713425442.638977 punch from 0x001fa0 to 0x00af6e (0x8fcf bytes) 000959 1713425442.640175 read 0x003085 thru 0x011f52 (0xeece bytes) 000960 1713425442.642168 mapwrite 0x00944d thru 0x00b4de (0x2092 bytes) 000961 1713425442.647840 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000962 1713425442.689951 read 0x00d67f thru 0x00dd93 (0x715 bytes) 000963 1713425442.691595 write 0x01e0d6 thru 0x0248cc (0x67f7 bytes) 000964 1713425442.693100 write 0x017533 thru 0x026b9f (0xf66d bytes) 000965 1713425442.695010 trunc from 0x026ba0 to 0x01bf02 (0xac9d bytes) 000966 1713425442.698654 punch from 0x01062a to 0x015ba4 (0x557b bytes) 000967 1713425442.699929 punch from 0x011938 to 0x01bf02 (0xa5cb bytes) 000968 1713425442.703103 punch from 0x012df3 to 0x01829e (0x54ac bytes) 000970 1713425442.704705 mapread 0x003a3d thru 0x007ee4 (0x44a8 bytes) 000971 1713425442.708236 trunc from 0x01bf03 to 0x01d06d (0x116b bytes) 000972 1713425442.711734 write 0x027534 thru 0x034267 (0xcd34 bytes) 000973 1713425442.712969 mapwrite 0x02eabb thru 0x037f0c (0x9452 bytes) 000974 1713425442.723210 mapwrite 0x02ff02 thru 0x035ebe (0x5fbd bytes) 000975 1713425442.727910 mapread 0x020c73 thru 0x0297ef (0x8b7d bytes) 000976 1713425442.730231 fallocate 0x00683c thru 0x0090c2 (0x2887 bytes) 000977 1713425442.731483 read 0x0254b5 thru 0x028a38 (0x3584 bytes) 000978 1713425442.731739 trunc from 0x037f0d to 0x01614b (0x21dc1 bytes) 000979 1713425442.735771 mapwrite 0x034484 thru 0x03bb62 (0x76df bytes) 000980 1713425442.747315 write 0x0282d8 thru 0x02f7ae (0x74d7 bytes) 000981 1713425442.747970 read 0x01eb2d thru 0x022ccc (0x41a0 bytes) 000983 1713425442.748560 write 0x033831 thru 0x03ffff (0xc7cf bytes) 000985 1713425442.749524 write 0x00f58f thru 0x019e3b (0xa8ad bytes) 000986 1713425442.750281 write 0x030f26 thru 0x039c04 (0x8cdf bytes) 000987 1713425442.751150 fallocate 0x029c2b thru 0x02e19a (0x4570 bytes) 000988 1713425442.752716 trunc from 0x040000 to 0x01f54f (0x20ab0 bytes) 000989 1713425442.757970 MIRROR_EXTEND to 12 mirrors 000991 1713425442.881526 write 0x034c72 thru 0x03ce6e (0x81fd bytes) 000992 1713425442.892580 read 0x0199ba thru 0x022ff5 (0x963c bytes) 000994 1713425442.895437 fallocate 0x02419b thru 0x02dbca (0x9a30 bytes) 000996 1713425442.897524 punch from 0x017148 to 0x01907f (0x1f38 bytes) 000997 1713425442.899093 mapread 0x039aa1 thru 0x03ce6e (0x33ce bytes) 000998 1713425442.899637 trunc from 0x03ce6f to 0x0176ac (0x257c2 bytes) 000999 1713425442.908639 mapread 0x005c7d thru 0x0060fe (0x482 bytes) 001000 1713425442.911372 write 0x024ea3 thru 0x034b8e (0xfcec bytes) All operations completed A-OK! PASS 70a (15s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 03:30:45 (1713425445) Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre2 Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6651360 .. Extending file size to 5267872 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7523872 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8088512 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8776512 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9192736 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8835488 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9288640 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9207680 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 12443 12444 12445 12447 12450 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:31:50 (1713425510) Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre2 Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2478432 .. Extending file size to 2184672 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7442368 .. Extending file size to 4408352 .. Extending file size to 7915680 .. Extending file size to 7707136 .. failed Extending file size to 8322656 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7919520 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8663968 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9014880 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8363232 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8380896 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8730048 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8836736 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8909856 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9126528 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 15867 15868 15869 15871 15873 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:32:55 (1713425575) Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3752 1283936 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2168 1285520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18776 3588244 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18528 3588492 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37304 7176736 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3752 1283936 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2168 1285520 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18776 3588244 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18528 3588492 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37304 7176736 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0354923 s, 59.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0518196 s, 40.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:33:01 (1713425581) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0245183 s, 85.5 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 03:33:04 (1713425584) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x100c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x100c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x100d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x100c:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:33:07 (1713425587) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0237757 s, 88.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x100e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x100d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x100e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:33:10 (1713425590) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0240469 s, 87.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x100f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1010:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x100f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x100f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1010:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x100f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1010:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:33:14 (1713425594) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0187845 s, 112 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } PASS 204c (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:33:38 (1713425618) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148314 s, 70.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } PASS 204d (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:34:12 (1713425652) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1011:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1012:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0211337 s, 99.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1011:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1012:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1011:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1011:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1012:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1011:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:34:47 (1713425687) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool oleg303-server: Pool lustre.test_204f created Adding targets to pool oleg303-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg303-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0204258 s, 103 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg303-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg303-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg303-server: Pool lustre.test_204f destroyed PASS 204f (40s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 03:35:29 (1713425729) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:35:33 (1713425733) oleg303-server: Pool lustre.test_206 created Adding targets to pool oleg303-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg303-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg303-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg303-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg303-server: Pool lustre.test_206 destroyed PASS 206 (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:35:48 (1713425748) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184333 s, 56.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129049 s, 81.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x81e32250 Mirror 3: 0x81e32250 PASS 207 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:35:53 (1713425753) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:35:56 (1713425756) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:35:58 (1713425758) Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00968919 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1390000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x760000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x760000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x760000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23a0000 Read this (4 bytes): BBBB 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:36:15 (1713425775) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00273083 s, 3.0 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 03:36:19 (1713425779) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116492 s, 90.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84b lmm_fid: [0x200000405:0x84b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84b lmm_fid: [0x200000405:0x84b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x101b:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339613259 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-18 03:36:19.000000000 -0400 Modify: 2024-04-18 03:36:19.000000000 -0400 Change: 2024-04-18 03:36:19.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:36:23 (1713425783) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132001 s, 79.4 MB/s oleg303-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg303-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg303-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg303-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg303-server: O/280000401/d28/101c: File not found by ext2_lookup PASS 210b (10s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 783 sec ============ 03:36:34 (1713425794) === sanity-flr: start cleanup 03:36:35 (1713425795) === === sanity-flr: finish cleanup 03:36:36 (1713425796) ===