-----============= acceptance-small: sanity-flr ============----- Thu Apr 18 20:16:40 EDT 2024 excepting tests: 6 201 44c === sanity-flr: start setup 20:16:44 (1713485804) === oleg119-client.virtnet: executing check_config_client /mnt/lustre oleg119-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg119-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b636e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b636e800.idle_timeout=debug disable quota as required oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:16:52 (1713485812) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7533] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 20:16:54 (1713485814) PASS 0a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:17:00 (1713485820) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:17:03 (1713485823) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 20:17:06 (1713485826) PASS 0d (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:17:13 (1713485833) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:17:16 (1713485836) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 20:17:18 (1713485838) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b636e800.stats=clear osc.lustre-OST0001-osc-ffff8800b636e800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:17:23 (1713485843) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:17:28 (1713485848) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:17:32 (1713485852) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 20:17:35 (1713485855) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 20:17:39 (1713485859) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:17:43 (1713485863) PASS 3 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:17:46 (1713485866) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:17:50 (1713485870) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 20:17:55 (1713485875) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:17:59 (1713485879) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.567513 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.03352 s, 31.3 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:18:04 (1713485884) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0209164 s, 100 MB/s PASS 22 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:18:07 (1713485887) Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b636e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b636e800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b636e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b636e800.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:18:33 (1713485913) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0865142 s, 60.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b636e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b636e800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:18:47 (1713485927) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b636e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b636e800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b636e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b636e800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:19:12 (1713485952) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.796057 s, 65.9 MB/s -rw-r--r-- 1 root root 52428800 Apr 18 20:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.218s user 0m0.000s sys 0m0.109s Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b636e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b636e800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.231s user 0m0.000s sys 0m0.116s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b636e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b636e800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.250s user 0m0.001s sys 0m0.115s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg119-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b636e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b636e800.ost_server_uuid in FULL state after 0 sec PASS 33b (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:19:38 (1713485978) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:19:41 (1713485981) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:19:44 (1713485984) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 20:19:46 (1713485986) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 20:19:50 (1713485990) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.165824 s, 75.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:19:55 (1713485995) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106185 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00979588 s, 107 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:19:59 (1713485999) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:20:03 (1713486003) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000832961 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000665331 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000625011 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000714906 s, 1.4 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 20:20:07 (1713486007) create test files with size 15754493 .. 9be432f800ec052879a8e34dd2844e51 - 34971299f54985850b78a0d9554fa594 - eaa28f5d7b7b8ec3b3faab9272e948d7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre PASS 37 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 20:20:15 (1713486015) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:20:19 (1713486019) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:20:23 (1713486023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196069 s, 53.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101999 s, 103 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 20:20:26 (1713486026) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0234786 s, 89.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0847824 s, 49.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0349591 s, 60.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 20:20:30 (1713486030) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 20:20:33 (1713486033) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:20:35 (1713486035) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114234 s, 91.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 20:20:39 (1713486039) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.082292 s, 38.2 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:20:43 (1713486043) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x45:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:20:47 (1713486047) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.076278 s, 137 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:20:51 (1713486051) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b6e16800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0692467 s, 151 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff880136a6b000.xattr_cache=1 PASS 44e (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:20:56 (1713486056) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 20:21:05 (1713486065) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 20:21:14 (1713486074) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 20:21:17 (1713486077) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0914946 s, 34.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0564285 s, 55.7 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: b8cf73ea4eecc43fc55fed47eac21871 - mirror 1 checksum: 77193dad19b455793ad128cfbafea1e6 - mirror 2 checksum: b8cf73ea4eecc43fc55fed47eac21871 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 20:21:18 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 20:21:18 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b8cf73ea4eecc43fc55fed47eac21871 - mirror 1 checksum: 77193dad19b455793ad128cfbafea1e6 - mirror 2 checksum: 77193dad19b455793ad128cfbafea1e6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 20:21:18 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 20:21:18 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 48 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:21:22 (1713486082) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00880932 s, 119 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:21:25 (1713486085) Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 20:21:33 (1713486093) shut down Failover mds1 to oleg119-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 20:21:46 (1713486106) targets are mounted 20:21:46 (1713486106) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116582 s, 89.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 50A (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:21:55 (1713486115) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0121113 s, 1.7 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00423558 s, 4.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 20:21:59 (1713486119) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0118849 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00335183 s, 6.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:22:03 (1713486123) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.0029358 s, 5.6 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 20:22:07 (1713486127) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0117106 s, 1.7 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:22:11 (1713486131) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00423729 s, 967 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0239416 s, 171 kB/s PASS 60a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:22:15 (1713486135) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:22:21 (1713486141) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:22:41 (1713486161) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:22:59 (1713486179) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:23:17 (1713486197) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 20:23:21 (1713486201) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 30628 mirror split pid 30629 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 30628 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 30629 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 20:24:24 (1713486264) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6265 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1713486265.126323 mapwrite 0x00e997 thru 0x00ee46 (0x4b0 bytes) 000003 1713486265.142229 write 0x020f40 thru 0x02e05f (0xd120 bytes) 000005 1713486265.143661 write 0x02a4db thru 0x030d81 (0x68a7 bytes) truncating to largest ever: 0x1da4d 000007 1713486265.144658 trunc from 0x030d82 to 0x01da4c (0x13335 bytes) 000008 1713486265.150528 fallocate 0x00f841 thru 0x018f97 (0x9757 bytes) 000009 1713486265.152598 write 0x01e9a8 thru 0x02cb71 (0xe1ca bytes) 000010 1713486265.153738 write 0x019ab4 thru 0x02908b (0xf5d8 bytes) fallocating to largest ever: 0x307c9 000014 1713486265.157439 fallocate 0x0227fc thru 0x0307c8 (0xdfcd bytes) 000015 1713486265.159591 write 0x00a0f6 thru 0x00b3dd (0x12e8 bytes) 000017 1713486265.163220 punch from 0x01ca46 to 0x02108d (0x4648 bytes) 000018 1713486265.169323 read 0x002edf thru 0x003d9c (0xebe bytes) 000019 1713486265.171469 mapread 0x00bf83 thru 0x018d44 (0xcdc2 bytes) fallocating to largest ever: 0x37813 000020 1713486265.174721 fallocate 0x02bc9e thru 0x037812 (0xbb75 bytes) 000021 1713486265.177088 read 0x0100d5 thru 0x012031 (0x1f5d bytes) 000022 1713486265.177477 punch from 0x01903d to 0x021101 (0x80c5 bytes) 000023 1713486265.179446 trunc from 0x0307c9 to 0x02ef8f (0x1839 bytes) 000024 1713486265.183869 write 0x009e66 thru 0x00cbee (0x2d89 bytes) 000025 1713486265.184601 trunc from 0x02ef90 to 0x0120f8 (0x1ce97 bytes) truncating to largest ever: 0x3c42a 000027 1713486265.189033 trunc from 0x0120f9 to 0x03c429 (0x2a331 bytes) 000028 1713486265.193887 MIRROR_EXTEND to 2 mirrors 000029 1713486265.329654 read 0x023384 thru 0x02cd9a (0x9a17 bytes) fallocating to largest ever: 0x40000 000030 1713486265.333764 fallocate 0x03294b thru 0x03ffff (0xd6b5 bytes) 000031 1713486265.345201 fallocate 0x0279fb thru 0x02b80d (0x3e13 bytes) 000032 1713486265.347255 trunc from 0x040000 to 0x03d8ee (0x2711 bytes) 000033 1713486265.359227 mapwrite 0x00dbdc thru 0x01458c (0x69b1 bytes) 000034 1713486265.376883 MIRROR_EXTEND to 3 mirrors 000036 1713486265.499472 write 0x0317ca thru 0x034ab2 (0x32e9 bytes) 000037 1713486265.513856 punch from 0x036ec2 to 0x03d8ee (0x6a2d bytes) 000038 1713486265.515935 read 0x010950 thru 0x016ec2 (0x6573 bytes) 000039 1713486265.518376 write 0x03e9d1 thru 0x03ffff (0x162f bytes) 000040 1713486265.519039 mapread 0x026862 thru 0x031f0b (0xb6aa bytes) 000041 1713486265.524208 write 0x02eef3 thru 0x033eb7 (0x4fc5 bytes) 000042 1713486265.525027 mapwrite 0x02945b thru 0x030149 (0x6cef bytes) 000043 1713486265.533458 mapwrite 0x038fa4 thru 0x03e74f (0x57ac bytes) 000044 1713486265.540978 mapwrite 0x03b9b0 thru 0x03ffff (0x4650 bytes) 000045 1713486265.546565 read 0x010655 thru 0x0131c7 (0x2b73 bytes) 000046 1713486265.547099 read 0x019f0b thru 0x02911e (0xf214 bytes) 000047 1713486265.550550 write 0x027516 thru 0x033d9b (0xc886 bytes) 000048 1713486265.553170 MIRROR_EXTEND to 4 mirrors 000049 1713486265.702201 fallocate 0x03d4d7 thru 0x03ffff (0x2b29 bytes) 000050 1713486265.717657 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000051 1713486265.771355 mapread 0x00eff0 thru 0x016625 (0x7636 bytes) 000052 1713486265.777292 mapread 0x01cca9 thru 0x021f8b (0x52e3 bytes) 000053 1713486265.782290 MIRROR_RESYNC 3 mirrors 000055 1713486265.955497 mapwrite 0x0210e8 thru 0x0224a0 (0x13b9 bytes) 000056 1713486265.990393 trunc from 0x040000 to 0x03cff0 (0x300f bytes) 000057 1713486266.004945 MIRROR_RESYNC 3 mirrors 000058 1713486266.243509 write 0x039b00 thru 0x03f530 (0x5a31 bytes) 000059 1713486266.254518 trunc from 0x03f531 to 0x016347 (0x291e9 bytes) 000060 1713486266.261889 MIRROR_RESYNC 3 mirrors 000061 1713486266.393715 read 0x005fce thru 0x00d5e2 (0x7615 bytes) 000062 1713486266.396580 read 0x00a854 thru 0x011a5b (0x7208 bytes) 000064 1713486266.399575 punch from 0x015e23 to 0x016191 (0x36f bytes) 000065 1713486266.411980 punch from 0x009742 to 0x00fd0f (0x65ce bytes) 000066 1713486266.414366 mapread 0x014724 thru 0x014a6e (0x34b bytes) 000067 1713486266.417338 write 0x007a38 thru 0x009a6e (0x2037 bytes) 000068 1713486266.421149 punch from 0x00fda9 to 0x016347 (0x659f bytes) 000070 1713486266.423745 trunc from 0x016348 to 0x013ac5 (0x2882 bytes) 000072 1713486266.434617 write 0x004b80 thru 0x00fc60 (0xb0e1 bytes) 000073 1713486266.439370 trunc from 0x013ac6 to 0x03c41d (0x28958 bytes) 000074 1713486266.447516 fallocate 0x015509 thru 0x01a3a9 (0x4ea1 bytes) 000075 1713486266.449807 mapwrite 0x00e3bf thru 0x013ba7 (0x57e9 bytes) 000076 1713486266.471592 punch from 0x03124e to 0x03c41d (0xb1d0 bytes) 000077 1713486266.474503 write 0x036f12 thru 0x03fe0e (0x8efd bytes) 000078 1713486266.477063 write 0x00775a thru 0x015999 (0xe240 bytes) 000079 1713486266.481977 mapread 0x00d266 thru 0x01917b (0xbf16 bytes) 000081 1713486266.484547 mapwrite 0x018b52 thru 0x01949e (0x94d bytes) 000083 1713486266.493907 trunc from 0x03fe0f to 0x01f863 (0x205ab bytes) 000086 1713486266.501615 fallocate 0x0031d5 thru 0x008b9e (0x59ca bytes) 000087 1713486266.504114 punch from 0x00f5a0 to 0x01c146 (0xcba7 bytes) 000090 1713486266.509735 MIRROR_RESYNC 3 mirrors 000091 1713486266.661687 write 0x024d64 thru 0x02da72 (0x8d0f bytes) 000093 1713486266.675086 punch from 0x01cca6 to 0x02a975 (0xdcd0 bytes) 000094 1713486266.682039 mapread 0x002fa7 thru 0x012b42 (0xfb9c bytes) 000095 1713486266.696728 trunc from 0x02da73 to 0x03997c (0xbf0a bytes) 000096 1713486266.714743 trunc from 0x03997d to 0x018545 (0x21437 bytes) 000097 1713486266.731305 MIRROR_EXTEND to 4 mirrors 000098 1713486266.941879 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000099 1713486267.036164 fallocate 0x01455d thru 0x01caf1 (0x8595 bytes) 000100 1713486267.050080 fallocate 0x002f04 thru 0x00670c (0x3809 bytes) 000101 1713486267.052064 fallocate 0x02de1e thru 0x036d07 (0x8eea bytes) 000102 1713486267.054039 trunc from 0x036d08 to 0x03a3b8 (0x36b1 bytes) 000103 1713486267.060731 mapread 0x020045 thru 0x0296b9 (0x9675 bytes) 000104 1713486267.067607 write 0x02ca7a thru 0x02f0fa (0x2681 bytes) 000105 1713486267.068369 write 0x03f0cf thru 0x03ffff (0xf31 bytes) 000106 1713486267.068989 trunc from 0x040000 to 0x039c0b (0x63f4 bytes) 000108 1713486267.074703 mapwrite 0x033f78 thru 0x0342fe (0x387 bytes) 000109 1713486267.088339 write 0x032efa thru 0x038877 (0x597e bytes) 000110 1713486267.089151 write 0x0363e5 thru 0x03d731 (0x734d bytes) 000112 1713486267.092646 MIRROR_EXTEND to 4 mirrors 000114 1713486267.278681 mapread 0x024c2a thru 0x028dd9 (0x41b0 bytes) 000115 1713486267.294214 punch from 0x02da9b to 0x0379bc (0x9f22 bytes) 000116 1713486267.306886 punch from 0x028d47 to 0x03620a (0xd4c4 bytes) 000117 1713486267.311834 MIRROR_RESYNC 4 mirrors 000119 1713486267.590000 mapread 0x037e79 thru 0x03d731 (0x58b9 bytes) 000120 1713486267.596701 mapread 0x03604a thru 0x03d731 (0x76e8 bytes) 000121 1713486267.597225 read 0x02a1d8 thru 0x02b913 (0x173c bytes) 000122 1713486267.599743 read 0x013673 thru 0x01e1f9 (0xab87 bytes) 000123 1713486267.606617 fallocate 0x02b87c thru 0x0324f5 (0x6c7a bytes) 000124 1713486267.625187 write 0x0077a8 thru 0x00cadc (0x5335 bytes) 000125 1713486267.629734 write 0x014e5d thru 0x0242c1 (0xf465 bytes) 000126 1713486267.634299 read 0x01f787 thru 0x025eaa (0x6724 bytes) 000128 1713486267.639486 mapread 0x02f670 thru 0x031247 (0x1bd8 bytes) 000129 1713486267.640294 trunc from 0x03d732 to 0x000b5c (0x3cbd5 bytes) 000132 1713486267.648908 mapread 0x00007c thru 0x000b5c (0xae1 bytes) 000133 1713486267.649794 mapread 0x000906 thru 0x000b5c (0x257 bytes) 000134 1713486267.650357 mapread 0x0003f9 thru 0x000b5c (0x764 bytes) 000136 1713486267.651064 mapread 0x000a16 thru 0x000b5c (0x147 bytes) 000137 1713486267.651557 mapread 0x000af8 thru 0x000b5c (0x65 bytes) 000139 1713486267.652829 fallocate 0x007b1b thru 0x009447 (0x192d bytes) 000140 1713486267.655494 punch from 0x00534e to 0x005c93 (0x946 bytes) 000143 1713486267.657405 fallocate 0x0120ad thru 0x01add3 (0x8d27 bytes) 000145 1713486267.662332 write 0x03215c thru 0x035f0c (0x3db1 bytes) 000146 1713486267.663043 mapwrite 0x035f1e thru 0x03ffff (0xa0e2 bytes) 000147 1713486267.692122 write 0x02d4ba thru 0x03925a (0xbda1 bytes) 000148 1713486267.694624 write 0x018adf thru 0x01d384 (0x48a6 bytes) 000149 1713486267.697621 trunc from 0x040000 to 0x01bbe2 (0x2441d bytes) 000150 1713486267.703335 punch from 0x012c4c to 0x01bbe2 (0x8f97 bytes) 000151 1713486267.708109 write 0x023bf4 thru 0x0263e6 (0x27f3 bytes) 000152 1713486267.708949 trunc from 0x0263e7 to 0x039228 (0x12e42 bytes) 000153 1713486267.717119 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000154 1713486267.790670 read 0x009347 thru 0x00a698 (0x1352 bytes) 000156 1713486267.794068 mapread 0x032dbd thru 0x039228 (0x646c bytes) 000157 1713486267.803394 read 0x015309 thru 0x022dd9 (0xdad1 bytes) 000158 1713486267.811131 trunc from 0x039229 to 0x000da9 (0x3847f bytes) 000161 1713486267.822438 fallocate 0x004d96 thru 0x0102df (0xb54a bytes) 000162 1713486267.824834 fallocate 0x014ed8 thru 0x01e678 (0x97a1 bytes) 000163 1713486267.829830 MIRROR_RESYNC 3 mirrors 000164 1713486267.944030 mapwrite 0x019560 thru 0x022fc2 (0x9a63 bytes) 000165 1713486267.972121 mapwrite 0x008aec thru 0x00a7c2 (0x1cd7 bytes) 000166 1713486267.981173 mapwrite 0x012ab4 thru 0x01c674 (0x9bc1 bytes) 000167 1713486267.991862 mapwrite 0x01c5a2 thru 0x0255ec (0x904b bytes) 000168 1713486268.011838 fallocate 0x027c4b thru 0x02f47c (0x7832 bytes) 000169 1713486268.014670 read 0x01fb0c thru 0x0255ec (0x5ae1 bytes) 000171 1713486268.015685 mapread 0x008774 thru 0x013524 (0xadb1 bytes) 000172 1713486268.018967 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000173 1713486268.089073 fallocate 0x005ad5 thru 0x00b412 (0x593e bytes) 000174 1713486268.091344 mapwrite 0x03be3c thru 0x03d1e1 (0x13a6 bytes) 000175 1713486268.109897 mapwrite 0x02b1c9 thru 0x02f83d (0x4675 bytes) 000176 1713486268.118956 mapread 0x01a5eb thru 0x028cb6 (0xe6cc bytes) 000177 1713486268.124902 MIRROR_RESYNC 2 mirrors 000178 1713486268.295820 punch from 0x0166a3 to 0x026234 (0xfb92 bytes) 000179 1713486268.311455 read 0x0395ab thru 0x03d1e1 (0x3c37 bytes) 000180 1713486268.315356 trunc from 0x03d1e2 to 0x027f6c (0x15275 bytes) 000181 1713486268.331776 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000182 1713486268.379583 mapwrite 0x031996 thru 0x0397dc (0x7e47 bytes) 000183 1713486268.410977 punch from 0x00062b to 0x003cc7 (0x369d bytes) 000184 1713486268.412771 read 0x033ed0 thru 0x035bae (0x1cdf bytes) 000185 1713486268.413189 mapwrite 0x03621d thru 0x03ffff (0x9de3 bytes) 000186 1713486268.431537 punch from 0x00f808 to 0x012b03 (0x32fc bytes) 000187 1713486268.434072 punch from 0x00d305 to 0x012dac (0x5aa8 bytes) 000188 1713486268.435878 write 0x009b17 thru 0x017902 (0xddec bytes) 000189 1713486268.441242 read 0x03cce6 thru 0x03ffff (0x331a bytes) 000190 1713486268.441689 trunc from 0x040000 to 0x020812 (0x1f7ed bytes) 000191 1713486268.447321 read 0x01ca03 thru 0x020812 (0x3e10 bytes) 000192 1713486268.451084 mapread 0x015e98 thru 0x01b10b (0x5274 bytes) 000193 1713486268.451808 read 0x00b2fd thru 0x014abc (0x97c0 bytes) 000194 1713486268.452239 mapread 0x01edc5 thru 0x020812 (0x1a4e bytes) 000196 1713486268.453110 write 0x016389 thru 0x0167cb (0x443 bytes) 000197 1713486268.453514 read 0x006559 thru 0x006d6d (0x815 bytes) 000199 1713486268.454227 read 0x01ffcd thru 0x020812 (0x846 bytes) 000200 1713486268.454589 fallocate 0x02bd34 thru 0x03bca8 (0xff75 bytes) 000202 1713486268.457245 punch from 0x010602 to 0x019ecd (0x98cc bytes) 000203 1713486268.462864 mapread 0x02664a thru 0x02a245 (0x3bfc bytes) 000204 1713486268.467270 fallocate 0x019a67 thru 0x020e5f (0x73f9 bytes) 000205 1713486268.469101 punch from 0x013d2d to 0x019813 (0x5ae7 bytes) 000206 1713486268.470757 trunc from 0x03bca9 to 0x014eeb (0x26dbd bytes) 000207 1713486268.475367 fallocate 0x02ebb5 thru 0x031e6b (0x32b7 bytes) 000208 1713486268.477229 read 0x021065 thru 0x028614 (0x75b0 bytes) 000209 1713486268.480108 mapread 0x017937 thru 0x0227ac (0xae76 bytes) 000210 1713486268.480616 trunc from 0x031e6c to 0x021e81 (0xffea bytes) 000213 1713486268.485700 punch from 0x016d6d to 0x021e81 (0xb115 bytes) 000214 1713486268.487752 punch from 0x00c09a to 0x00d0f2 (0x1059 bytes) 000215 1713486268.489390 read 0x01085f thru 0x01299d (0x213f bytes) 000216 1713486268.489764 punch from 0x00ffa2 to 0x01a9c8 (0xaa27 bytes) 000217 1713486268.491542 punch from 0x011b52 to 0x01fcb0 (0xe15f bytes) 000221 1713486268.493535 mapread 0x013872 thru 0x02116e (0xd8fd bytes) 000224 1713486268.496736 mapread 0x020d08 thru 0x021e81 (0x117a bytes) 000226 1713486268.497308 mapread 0x00391e thru 0x00cd70 (0x9453 bytes) 000227 1713486268.497774 mapwrite 0x02cc1b thru 0x030111 (0x34f7 bytes) 000230 1713486268.513693 punch from 0x015570 to 0x01c11c (0x6bad bytes) 000231 1713486268.516088 trunc from 0x030112 to 0x00e9bd (0x21754 bytes) 000232 1713486268.522226 trunc from 0x00e9be to 0x010ef0 (0x2533 bytes) 000233 1713486268.528581 fallocate 0x02da21 thru 0x033acc (0x60ac bytes) 000235 1713486268.530991 read 0x0087ff thru 0x00d0c1 (0x48c3 bytes) 000237 1713486268.531671 mapread 0x0097d1 thru 0x010ef0 (0x7720 bytes) 000238 1713486268.534238 mapread 0x00bf16 thru 0x010ef0 (0x4fdb bytes) 000239 1713486268.534808 fallocate 0x02d59c thru 0x02f8be (0x2323 bytes) 000240 1713486268.537096 mapwrite 0x012f0e thru 0x021177 (0xe26a bytes) 000241 1713486268.558737 read 0x01523c thru 0x0167dd (0x15a2 bytes) 000242 1713486268.559047 write 0x010c82 thru 0x01ab16 (0x9e95 bytes) 000244 1713486268.559980 trunc from 0x021178 to 0x02d0ed (0xbf76 bytes) 000245 1713486268.564591 trunc from 0x02d0ee to 0x03ab69 (0xda7c bytes) 000246 1713486268.569730 punch from 0x031ece to 0x03ab69 (0x8c9c bytes) 000248 1713486268.573349 MIRROR_EXTEND to 2 mirrors 000251 1713486268.756348 write 0x01d253 thru 0x0237d0 (0x657e bytes) 000252 1713486268.773849 mapread 0x015ae5 thru 0x01a549 (0x4a65 bytes) 000253 1713486268.778908 punch from 0x0351c1 to 0x03ab69 (0x59a9 bytes) 000255 1713486268.782503 mapread 0x033ef3 thru 0x037e8f (0x3f9d bytes) 000257 1713486268.786727 mapread 0x0270ed thru 0x0281da (0x10ee bytes) 000258 1713486268.792088 read 0x02ba1c thru 0x0303b9 (0x499e bytes) 000260 1713486268.793075 trunc from 0x03ab6a to 0x00a47b (0x306ee bytes) 000262 1713486268.812934 read 0x007275 thru 0x00a47b (0x3207 bytes) 000263 1713486268.816878 write 0x028b79 thru 0x031002 (0x848a bytes) 000264 1713486268.817872 write 0x03974a thru 0x03e2a7 (0x4b5e bytes) 000265 1713486268.818633 read 0x00cf68 thru 0x011340 (0x43d9 bytes) 000266 1713486268.823195 read 0x0009f9 thru 0x005116 (0x471e bytes) 000267 1713486268.823651 read 0x035ea1 thru 0x03e2a7 (0x8407 bytes) 000269 1713486268.824369 fallocate 0x03bdd2 thru 0x03ffff (0x422e bytes) 000270 1713486268.827245 fallocate 0x025264 thru 0x02b74f (0x64ec bytes) 000271 1713486268.829547 read 0x02e70b thru 0x0353fd (0x6cf3 bytes) 000273 1713486268.832170 MIRROR_EXTEND to 3 mirrors 000276 1713486269.056408 punch from 0x018805 to 0x01f322 (0x6b1e bytes) 000278 1713486269.072624 trunc from 0x040000 to 0x021eaa (0x1e155 bytes) 000280 1713486269.087405 write 0x01444f thru 0x023fce (0xfb80 bytes) 000282 1713486269.090319 read 0x02347c thru 0x023fce (0xb53 bytes) 000283 1713486269.090721 fallocate 0x019dbf thru 0x0212ce (0x7510 bytes) 000284 1713486269.092892 fallocate 0x0048ad thru 0x00df60 (0x96b4 bytes) 000285 1713486269.094634 mapread 0x005674 thru 0x00843d (0x2dca bytes) 000286 1713486269.098843 write 0x008d86 thru 0x00dbbc (0x4e37 bytes) 000287 1713486269.099578 read 0x002bad thru 0x009e9f (0x72f3 bytes) 000288 1713486269.099938 punch from 0x012136 to 0x016f04 (0x4dcf bytes) 000289 1713486269.105798 read 0x0193dd thru 0x0235da (0xa1fe bytes) 000292 1713486269.106869 punch from 0x002ddf to 0x00625a (0x347c bytes) 000293 1713486269.109433 punch from 0x01807e to 0x020603 (0x8586 bytes) 000294 1713486269.112307 mapread 0x01e7f3 thru 0x01fa7d (0x128b bytes) 000295 1713486269.121595 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000296 1713486269.198204 read 0x018bbb thru 0x018d4e (0x194 bytes) 000298 1713486269.202290 mapread 0x0048af thru 0x00972b (0x4e7d bytes) 000299 1713486269.212950 mapwrite 0x01f10a thru 0x0223a8 (0x329f bytes) 000300 1713486269.232128 write 0x0138a5 thru 0x01f361 (0xbabd bytes) 000301 1713486269.239677 mapread 0x010f02 thru 0x014a69 (0x3b68 bytes) 000302 1713486269.240460 fallocate 0x0191b3 thru 0x01ba18 (0x2866 bytes) 000303 1713486269.243353 mapread 0x0048a1 thru 0x00729d (0x29fd bytes) 000304 1713486269.243885 mapwrite 0x0192ab thru 0x026e54 (0xdbaa bytes) 000306 1713486269.262519 mapwrite 0x003ebc thru 0x00fdf6 (0xbf3b bytes) 000307 1713486269.272488 MIRROR_RESYNC 2 mirrors 000308 1713486269.407980 fallocate 0x030463 thru 0x038831 (0x83cf bytes) 000310 1713486269.419230 fallocate 0x01da3e thru 0x029cce (0xc291 bytes) 000311 1713486269.421107 punch from 0x02ba44 to 0x036d02 (0xb2bf bytes) 000313 1713486269.425358 MIRROR_EXTEND to 3 mirrors 000314 1713486269.610932 write 0x01741c thru 0x022334 (0xaf19 bytes) 000315 1713486269.624982 fallocate 0x01de0f thru 0x01e7dd (0x9cf bytes) 000316 1713486269.628634 MIRROR_EXTEND to 4 mirrors 000317 1713486269.800006 mapwrite 0x02f764 thru 0x03e4e9 (0xed86 bytes) 000318 1713486269.843334 read 0x03c715 thru 0x03e4e9 (0x1dd5 bytes) 000319 1713486269.843827 mapwrite 0x00cf5b thru 0x0114fb (0x45a1 bytes) 000320 1713486269.856706 MIRROR_RESYNC 4 mirrors 000321 1713486270.153305 trunc from 0x03e4ea to 0x03236b (0xc17e bytes) 000322 1713486270.167199 trunc from 0x03236c to 0x01bc0c (0x1675f bytes) 000324 1713486270.173477 fallocate 0x01c9c3 thru 0x024052 (0x7690 bytes) 000325 1713486270.175292 trunc from 0x01bc0d to 0x039ff8 (0x1e3ec bytes) 000329 1713486270.179908 write 0x001183 thru 0x004e60 (0x3cde bytes) 000330 1713486270.183399 punch from 0x0389fb to 0x039ff8 (0x15fe bytes) 000331 1713486270.185130 mapread 0x01b0e4 thru 0x027759 (0xc676 bytes) 000332 1713486270.191003 mapread 0x025ee5 thru 0x02bc44 (0x5d60 bytes) 000333 1713486270.191537 read 0x02d1c0 thru 0x033c94 (0x6ad5 bytes) 000334 1713486270.193680 read 0x0134ed thru 0x019d54 (0x6868 bytes) 000335 1713486270.196665 trunc from 0x039ff9 to 0x03f4e4 (0x54ec bytes) 000336 1713486270.201143 trunc from 0x03f4e5 to 0x03d2f1 (0x21f3 bytes) 000337 1713486270.205297 write 0x0139ae thru 0x020808 (0xce5b bytes) 000338 1713486270.206321 mapread 0x032573 thru 0x03bab6 (0x9544 bytes) 000339 1713486270.208536 fallocate 0x0394e8 thru 0x03bec5 (0x29de bytes) 000340 1713486270.210276 mapwrite 0x03af3f thru 0x03ffff (0x50c1 bytes) 000341 1713486270.225383 punch from 0x006f59 to 0x013c90 (0xcd38 bytes) 000342 1713486270.228207 read 0x034d00 thru 0x038b67 (0x3e68 bytes) 000343 1713486270.228646 write 0x01a81a thru 0x020bdf (0x63c6 bytes) 000344 1713486270.229431 mapwrite 0x026259 thru 0x0359c2 (0xf76a bytes) 000346 1713486270.239185 read 0x00ea8c thru 0x01b4ee (0xca63 bytes) 000347 1713486270.242842 write 0x03d11f thru 0x03f35b (0x223d bytes) 000348 1713486270.245655 MIRROR_EXTEND to 5 mirrors 000349 1713486270.430186 punch from 0x00ac3e to 0x01036e (0x5731 bytes) 000350 1713486270.441420 trunc from 0x040000 to 0x01323f (0x2cdc0 bytes) 000351 1713486270.454782 trunc from 0x013240 to 0x026e76 (0x13c37 bytes) 000352 1713486270.461197 MIRROR_RESYNC 5 mirrors 000353 1713486270.682644 MIRROR_RESYNC 5 mirrors 000355 1713486270.728131 trunc from 0x026e77 to 0x03f7ee (0x18978 bytes) 000356 1713486270.745087 write 0x0241e5 thru 0x02fb98 (0xb9b4 bytes) 000357 1713486270.749212 write 0x002fd8 thru 0x0064bd (0x34e6 bytes) 000358 1713486270.752877 mapwrite 0x008045 thru 0x015b2a (0xdae6 bytes) 000361 1713486270.764659 mapwrite 0x01b0d6 thru 0x0280de (0xd009 bytes) 000362 1713486270.779843 fallocate 0x01b946 thru 0x027fa4 (0xc65f bytes) 000363 1713486270.782213 read 0x0115a2 thru 0x018a76 (0x74d5 bytes) 000364 1713486270.782630 trunc from 0x03f7ef to 0x00c34c (0x334a2 bytes) 000365 1713486270.789306 mapread 0x00a8ea thru 0x00c34c (0x1a63 bytes) 000366 1713486270.789974 mapread 0x00a83f thru 0x00c34c (0x1b0e bytes) 000367 1713486270.790495 punch from 0x009b18 to 0x00c34c (0x2835 bytes) 000368 1713486270.792371 mapread 0x006a58 thru 0x00c34c (0x58f5 bytes) 000369 1713486270.794783 punch from 0x00b0e7 to 0x00c34c (0x1266 bytes) 000373 1713486270.799559 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000374 1713486270.846278 mapwrite 0x0315ff thru 0x03aae5 (0x94e7 bytes) 000375 1713486270.864405 fallocate 0x01f369 thru 0x02d2fc (0xdf94 bytes) 000376 1713486270.866390 write 0x0219fc thru 0x025be9 (0x41ee bytes) 000377 1713486270.869543 fallocate 0x01067a thru 0x013787 (0x310e bytes) 000378 1713486270.871398 read 0x03707d thru 0x03aae5 (0x3a69 bytes) 000379 1713486270.871793 read 0x01f7f7 thru 0x02a392 (0xab9c bytes) 000380 1713486270.875863 punch from 0x0382ec to 0x03aae5 (0x27fa bytes) 000381 1713486270.878103 write 0x00b43b thru 0x011d38 (0x68fe bytes) 000382 1713486270.878862 punch from 0x02346a to 0x02cdb8 (0x994f bytes) 000383 1713486270.884281 trunc from 0x03aae6 to 0x0373c4 (0x3721 bytes) 000384 1713486270.889880 punch from 0x00e4a2 to 0x00e4d2 (0x31 bytes) 000387 1713486270.892210 write 0x00b997 thru 0x018704 (0xcd6e bytes) 000388 1713486270.893169 mapread 0x01aec1 thru 0x01c11e (0x125e bytes) 000389 1713486270.893663 punch from 0x010419 to 0x01e593 (0xe17b bytes) 000390 1713486270.899940 read 0x023405 thru 0x0266cd (0x32c9 bytes) 000391 1713486270.905207 MIRROR_EXTEND to 5 mirrors 000392 1713486271.071961 MIRROR_EXTEND to 6 mirrors 000394 1713486271.231805 mapwrite 0x02b871 thru 0x03197e (0x610e bytes) 000395 1713486271.269185 punch from 0x01609b to 0x01d76b (0x76d1 bytes) 000397 1713486271.271876 punch from 0x01776a to 0x01f54c (0x7de3 bytes) 000398 1713486271.274066 read 0x003ac3 thru 0x00bfd5 (0x8513 bytes) 000399 1713486271.280562 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000400 1713486271.342610 read 0x01876a thru 0x01e9d1 (0x6268 bytes) 000401 1713486271.346103 trunc from 0x0373c5 to 0x008ef3 (0x2e4d1 bytes) 000402 1713486271.354086 punch from 0x005599 to 0x008ef3 (0x395b bytes) 000403 1713486271.356205 mapread 0x008a68 thru 0x008ef3 (0x48c bytes) 000405 1713486271.362096 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000406 1713486271.415501 read 0x003e54 thru 0x007b22 (0x3ccf bytes) 000407 1713486271.418765 write 0x022feb thru 0x02958b (0x65a1 bytes) 000408 1713486271.423255 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000409 1713486271.475783 mapwrite 0x01b660 thru 0x01c621 (0xfc2 bytes) 000412 1713486271.485049 write 0x03b4c5 thru 0x03ffff (0x4b3b bytes) 000413 1713486271.488992 MIRROR_EXTEND to 4 mirrors 000415 1713486271.650030 MIRROR_RESYNC 4 mirrors 000416 1713486271.826108 fallocate 0x037da7 thru 0x03ffff (0x8259 bytes) 000417 1713486271.835393 mapread 0x025783 thru 0x02fbe7 (0xa465 bytes) 000418 1713486271.842695 mapread 0x0019be thru 0x00f391 (0xd9d4 bytes) 000423 1713486271.848697 trunc from 0x040000 to 0x0328e5 (0xd71a bytes) 000424 1713486271.860621 fallocate 0x034fd8 thru 0x03ea14 (0x9a3d bytes) 000425 1713486271.862736 fallocate 0x01eac1 thru 0x0272af (0x87ef bytes) 000426 1713486271.864689 read 0x002a33 thru 0x00e286 (0xb854 bytes) 000427 1713486271.865084 trunc from 0x0328e6 to 0x0257db (0xd10a bytes) 000428 1713486271.868942 read 0x024d7f thru 0x0257db (0xa5d bytes) 000429 1713486271.869206 mapread 0x002c97 thru 0x007d2f (0x5099 bytes) 000430 1713486271.870774 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000431 1713486271.916752 fallocate 0x0087e1 thru 0x016482 (0xdca2 bytes) 000432 1713486271.918258 read 0x0139ae thru 0x01cba4 (0x91f7 bytes) 000434 1713486271.920899 trunc from 0x0257dc to 0x00e309 (0x174d2 bytes) 000435 1713486271.926200 fallocate 0x03549d thru 0x039847 (0x43ab bytes) 000436 1713486271.927946 mapread 0x0014c4 thru 0x008af9 (0x7636 bytes) 000437 1713486271.932715 fallocate 0x0199b5 thru 0x01b079 (0x16c5 bytes) 000438 1713486271.935908 MIRROR_EXTEND to 4 mirrors 000439 1713486272.076196 read 0x011f37 thru 0x0157ee (0x38b8 bytes) 000441 1713486272.081864 trunc from 0x039848 to 0x01df34 (0x1b913 bytes) 000443 1713486272.094029 mapread 0x0156fa thru 0x01a072 (0x4979 bytes) 000444 1713486272.099364 mapread 0x01c71b thru 0x01df34 (0x181a bytes) 000445 1713486272.099737 read 0x01cc63 thru 0x01df34 (0x12d2 bytes) 000446 1713486272.100086 trunc from 0x01df35 to 0x02ab56 (0xcc22 bytes) 000447 1713486272.104522 mapwrite 0x02ac5a thru 0x03771b (0xcac2 bytes) 000448 1713486272.119824 write 0x003c82 thru 0x008f01 (0x5280 bytes) 000449 1713486272.120611 mapread 0x0213e8 thru 0x02b6b5 (0xa2ce bytes) 000450 1713486272.121219 fallocate 0x01a3cc thru 0x01db5a (0x378f bytes) 000451 1713486272.123849 fallocate 0x014632 thru 0x01a2c1 (0x5c90 bytes) 000452 1713486272.125776 mapread 0x036d75 thru 0x03771b (0x9a7 bytes) 000453 1713486272.126330 read 0x03735a thru 0x03771b (0x3c2 bytes) 000454 1713486272.126734 mapwrite 0x00e173 thru 0x01095c (0x27ea bytes) 000456 1713486272.132324 punch from 0x014ddb to 0x020dfa (0xc020 bytes) 000457 1713486272.133981 punch from 0x024d5b to 0x031e2d (0xd0d3 bytes) 000458 1713486272.135912 write 0x0216c0 thru 0x0238e9 (0x222a bytes) 000459 1713486272.136535 trunc from 0x03771c to 0x00a410 (0x2d30b bytes) 000461 1713486272.141466 fallocate 0x000233 thru 0x009e85 (0x9c53 bytes) 000463 1713486272.143433 trunc from 0x00a411 to 0x02d4e5 (0x230d5 bytes) 000464 1713486272.147514 fallocate 0x022f4b thru 0x02f00a (0xc0c0 bytes) 000465 1713486272.149084 fallocate 0x00f06d thru 0x0163e9 (0x737d bytes) 000466 1713486272.150415 read 0x00e87f thru 0x015200 (0x6982 bytes) 000468 1713486272.153202 punch from 0x01824d to 0x021d3d (0x9af1 bytes) 000469 1713486272.154655 write 0x00f0b6 thru 0x013608 (0x4553 bytes) 000470 1713486272.155341 punch from 0x02dd19 to 0x02f00a (0x12f2 bytes) 000471 1713486272.158194 MIRROR_RESYNC 4 mirrors 000472 1713486272.356332 punch from 0x022a8b to 0x0296b9 (0x6c2f bytes) 000473 1713486272.367476 fallocate 0x01ebe6 thru 0x02bc09 (0xd024 bytes) 000474 1713486272.369885 write 0x01c028 thru 0x022d54 (0x6d2d bytes) 000477 1713486272.374032 read 0x0255b8 thru 0x02f00a (0x9a53 bytes) 000479 1713486272.377707 write 0x02cda9 thru 0x030088 (0x32e0 bytes) 000480 1713486272.378478 mapwrite 0x00867a thru 0x017c73 (0xf5fa bytes) 000481 1713486272.393160 fallocate 0x01819e thru 0x01b79c (0x35ff bytes) 000482 1713486272.396171 mapread 0x00d5c9 thru 0x010684 (0x30bc bytes) 000483 1713486272.396730 punch from 0x024b0e to 0x030088 (0xb57b bytes) 000484 1713486272.398986 mapread 0x003ac2 thru 0x009318 (0x5857 bytes) 000486 1713486272.402157 read 0x02bb37 thru 0x030088 (0x4552 bytes) 000492 1713486272.406872 mapread 0x02e162 thru 0x030088 (0x1f27 bytes) 000494 1713486272.407755 trunc from 0x030089 to 0x01ae72 (0x15216 bytes) 000495 1713486272.419294 trunc from 0x01ae73 to 0x00d995 (0xd4dd bytes) 000496 1713486272.426878 MIRROR_EXTEND to 5 mirrors 000497 1713486272.527615 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000498 1713486272.586379 mapwrite 0x00340f thru 0x009d79 (0x696b bytes) 000499 1713486272.624825 mapread 0x0081c3 thru 0x00d995 (0x57d3 bytes) 000500 1713486272.625486 mapread 0x002dc6 thru 0x00d293 (0xa4ce bytes) 000501 1713486272.626054 trunc from 0x00d996 to 0x020b1d (0x13188 bytes) 000503 1713486272.637354 trunc from 0x020b1e to 0x007cdc (0x18e41 bytes) 000504 1713486272.641059 trunc from 0x007cdd to 0x02002d (0x18351 bytes) 000505 1713486272.644879 mapread 0x012340 thru 0x015337 (0x2ff8 bytes) 000506 1713486272.647696 read 0x01fd92 thru 0x02002d (0x29c bytes) 000507 1713486272.649227 punch from 0x013ac3 to 0x018816 (0x4d54 bytes) 000508 1713486272.650567 write 0x038c2a thru 0x03ffff (0x73d6 bytes) 000509 1713486272.651293 write 0x0336bc thru 0x0383e7 (0x4d2c bytes) 000510 1713486272.655302 MIRROR_RESYNC 4 mirrors 000511 1713486272.803574 mapread 0x01585e thru 0x024e9d (0xf640 bytes) 000513 1713486272.811884 punch from 0x00f7d2 to 0x01bd5c (0xc58b bytes) 000514 1713486272.820981 trunc from 0x040000 to 0x00f140 (0x30ebf bytes) 000515 1713486272.834264 MIRROR_EXTEND to 5 mirrors 000516 1713486272.938423 mapwrite 0x0015e7 thru 0x0021c8 (0xbe2 bytes) 000518 1713486272.973400 write 0x022b42 thru 0x02d742 (0xac01 bytes) 000519 1713486272.975800 write 0x00eb86 thru 0x019b00 (0xaf7b bytes) 000520 1713486272.979002 MIRROR_EXTEND to 6 mirrors 000521 1713486273.109131 punch from 0x0038b5 to 0x0094b5 (0x5c01 bytes) 000522 1713486273.119187 trunc from 0x02d743 to 0x0065ca (0x27178 bytes) 000523 1713486273.124498 read 0x0044ab thru 0x0065ca (0x2120 bytes) 000524 1713486273.126149 punch from 0x0003c9 to 0x0065ca (0x6202 bytes) 000525 1713486273.127464 mapwrite 0x009af7 thru 0x00ac45 (0x114f bytes) 000526 1713486273.137386 mapread 0x009563 thru 0x00ac45 (0x16e3 bytes) 000527 1713486273.139207 MIRROR_EXTEND to 7 mirrors 000529 1713486273.240758 trunc from 0x00ac46 to 0x0132ea (0x86a5 bytes) 000530 1713486273.254507 write 0x03eef1 thru 0x03ffff (0x110f bytes) 000531 1713486273.255234 mapwrite 0x032dbd thru 0x03fb2b (0xcd6f bytes) 000532 1713486273.269515 MIRROR_EXTEND to 8 mirrors 000534 1713486273.388071 mapread 0x01d506 thru 0x02c4f5 (0xeff0 bytes) 000536 1713486273.401684 MIRROR_RESYNC 8 mirrors 000538 1713486273.627565 read 0x00220c thru 0x0031f9 (0xfee bytes) 000539 1713486273.629111 fallocate 0x018325 thru 0x0213de (0x90ba bytes) 000540 1713486273.638867 fallocate 0x034e55 thru 0x03e2a7 (0x9453 bytes) 000541 1713486273.640176 mapread 0x01a1dc thru 0x025e7e (0xbca3 bytes) 000542 1713486273.646024 punch from 0x010e94 to 0x014f26 (0x4093 bytes) 000543 1713486273.647429 punch from 0x009ad3 to 0x00ff66 (0x6494 bytes) 000545 1713486273.650800 MIRROR_EXTEND to 9 mirrors 000546 1713486273.782996 mapread 0x00c43d thru 0x0109fc (0x45c0 bytes) 000547 1713486273.791889 trunc from 0x040000 to 0x012fdf (0x2d020 bytes) 000548 1713486273.807192 fallocate 0x013ea6 thru 0x018790 (0x48eb bytes) 000549 1713486273.808616 read 0x004c82 thru 0x004eff (0x27e bytes) 000550 1713486273.810050 trunc from 0x012fe0 to 0x0330c8 (0x200e9 bytes) 000551 1713486273.815521 punch from 0x00fa75 to 0x01e76c (0xecf8 bytes) 000552 1713486273.817222 read 0x031620 thru 0x0330c8 (0x1aa9 bytes) 000553 1713486273.821829 write 0x0064b2 thru 0x01138c (0xaedb bytes) 000554 1713486273.822986 mapread 0x006929 thru 0x00de72 (0x754a bytes) 000556 1713486273.823765 mapread 0x01221b thru 0x0175bc (0x53a2 bytes) 000557 1713486273.824346 trunc from 0x0330c9 to 0x023d2b (0xf39d bytes) 000559 1713486273.829582 mapwrite 0x02f3e7 thru 0x02f73c (0x356 bytes) 000560 1713486273.843138 read 0x00202f thru 0x0054ca (0x349c bytes) 000561 1713486273.843608 mapread 0x0051c2 thru 0x012861 (0xd6a0 bytes) 000562 1713486273.844267 trunc from 0x02f73d to 0x02b9f0 (0x3d4c bytes) 000563 1713486273.848865 mapwrite 0x025594 thru 0x02e490 (0x8efd bytes) 000564 1713486273.863096 trunc from 0x02e491 to 0x0198be (0x14bd2 bytes) 000565 1713486273.868144 MIRROR_RESYNC 9 mirrors 000566 1713486274.182211 mapread 0x007dae thru 0x0096e7 (0x193a bytes) 000567 1713486274.187602 trunc from 0x0198bf to 0x01ad7a (0x14bc bytes) 000568 1713486274.200632 MIRROR_RESYNC 9 mirrors 000569 1713486274.509230 write 0x00709d thru 0x0108cb (0x982f bytes) 000571 1713486274.518442 punch from 0x013c71 to 0x0165c0 (0x2950 bytes) 000572 1713486274.520759 MIRROR_EXTEND to 10 mirrors 000573 1713486274.647694 punch from 0x015987 to 0x016c38 (0x12b2 bytes) 000574 1713486274.660573 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000576 1713486274.707563 punch from 0x01a57e to 0x01ad7a (0x7fd bytes) 000577 1713486274.709118 mapwrite 0x0299fb thru 0x02eb3b (0x5141 bytes) 000578 1713486274.723734 read 0x01b876 thru 0x01d522 (0x1cad bytes) 000579 1713486274.725409 mapwrite 0x021b35 thru 0x0239ec (0x1eb8 bytes) 000580 1713486274.731322 punch from 0x009855 to 0x018f9a (0xf746 bytes) 000581 1713486274.732833 trunc from 0x02eb3c to 0x03c32f (0xd7f4 bytes) 000582 1713486274.738765 MIRROR_RESYNC 9 mirrors 000583 1713486275.137575 mapread 0x02302e thru 0x025677 (0x264a bytes) 000584 1713486275.141971 mapwrite 0x004608 thru 0x0052f2 (0xceb bytes) 000585 1713486275.171725 MIRROR_EXTEND to 10 mirrors 000586 1713486275.294481 read 0x02bf37 thru 0x02ef3b (0x3005 bytes) 000588 1713486275.299501 punch from 0x034a6d to 0x03c32f (0x78c3 bytes) 000589 1713486275.310177 MIRROR_EXTEND to 11 mirrors 000590 1713486275.457487 trunc from 0x03c330 to 0x00b591 (0x30d9e bytes) 000591 1713486275.472465 punch from 0x00128c to 0x005d18 (0x4a8d bytes) 000593 1713486275.474508 trunc from 0x00b592 to 0x01e1be (0x12c2d bytes) 000594 1713486275.479622 mapread 0x004092 thru 0x00cc92 (0x8c01 bytes) 000595 1713486275.484705 punch from 0x002fd2 to 0x008329 (0x5358 bytes) 000596 1713486275.486397 mapread 0x00ef85 thru 0x01a1e4 (0xb260 bytes) 000597 1713486275.489194 write 0x001b7a thru 0x003724 (0x1bab bytes) 000599 1713486275.491905 mapwrite 0x01a384 thru 0x026adb (0xc758 bytes) 000601 1713486275.505048 trunc from 0x026adc to 0x03acb8 (0x141dd bytes) 000602 1713486275.509042 punch from 0x0376c6 to 0x0394ea (0x1e25 bytes) 000603 1713486275.510502 fallocate 0x02751c thru 0x030ccd (0x97b2 bytes) 000605 1713486275.512183 fallocate 0x0317bd thru 0x03b8ed (0xa131 bytes) 000606 1713486275.513537 trunc from 0x03b8ee to 0x020d6d (0x1ab80 bytes) 000607 1713486275.517828 fallocate 0x02068e thru 0x02f279 (0xebec bytes) 000608 1713486275.519446 mapwrite 0x03dc34 thru 0x03ffff (0x23cc bytes) 000609 1713486275.529854 mapread 0x029462 thru 0x037773 (0xe312 bytes) 000610 1713486275.535056 trunc from 0x040000 to 0x004bbf (0x3b440 bytes) 000611 1713486275.539189 write 0x01641c thru 0x0234c6 (0xd0ab bytes) 000612 1713486275.540102 mapread 0x01a797 thru 0x0234c6 (0x8d30 bytes) 000613 1713486275.540545 mapwrite 0x0095c9 thru 0x0156f2 (0xc12a bytes) 000614 1713486275.551245 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000615 1713486275.604283 MIRROR_EXTEND to 11 mirrors 000616 1713486275.735612 read 0x009aa5 thru 0x00a01a (0x576 bytes) 000617 1713486275.739605 mapread 0x019db8 thru 0x0234c6 (0x970f bytes) 000618 1713486275.747960 read 0x00c511 thru 0x00c926 (0x416 bytes) 000619 1713486275.752032 punch from 0x00a2db to 0x015c08 (0xb92e bytes) 000620 1713486275.761380 write 0x032988 thru 0x03a881 (0x7efa bytes) 000621 1713486275.763168 trunc from 0x03a882 to 0x00a3cb (0x304b6 bytes) 000622 1713486275.767500 mapwrite 0x019221 thru 0x01bd60 (0x2b40 bytes) 000623 1713486275.778739 punch from 0x019c38 to 0x01bd60 (0x2129 bytes) 000624 1713486275.780280 punch from 0x0109e3 to 0x012123 (0x1741 bytes) 000625 1713486275.781397 mapread 0x005b99 thru 0x00f509 (0x9971 bytes) 000626 1713486275.792333 fallocate 0x01dd7f thru 0x02a234 (0xc4b6 bytes) 000627 1713486275.793683 mapread 0x002105 thru 0x00b7a1 (0x969d bytes) 000628 1713486275.797096 trunc from 0x02a235 to 0x03d6bd (0x13489 bytes) 000629 1713486275.800361 mapwrite 0x02f16b thru 0x03c74f (0xd5e5 bytes) 000631 1713486275.810100 MIRROR_RESYNC 11 mirrors 000632 1713486276.479009 write 0x004d6a thru 0x00e955 (0x9bec bytes) 000634 1713486276.489111 mapwrite 0x028014 thru 0x033c38 (0xbc25 bytes) 000635 1713486276.500399 fallocate 0x024040 thru 0x029e25 (0x5de6 bytes) 000637 1713486276.502374 trunc from 0x03d6be to 0x012ea8 (0x2a815 bytes) 000638 1713486276.508573 fallocate 0x03cc3e thru 0x03ffff (0x33c2 bytes) 000639 1713486276.510543 read 0x02b1ab thru 0x0372b5 (0xc10b bytes) 000640 1713486276.513441 punch from 0x027aa2 to 0x03359e (0xbafd bytes) 000641 1713486276.514989 read 0x03777c thru 0x03ea5c (0x72e1 bytes) 000642 1713486276.519151 mapwrite 0x01ea24 thru 0x025ce9 (0x72c6 bytes) 000643 1713486276.526828 mapread 0x023244 thru 0x031e3f (0xebfc bytes) 000646 1713486276.528150 mapwrite 0x005833 thru 0x010e98 (0xb666 bytes) 000647 1713486276.536209 write 0x006ebe thru 0x013413 (0xc556 bytes) 000648 1713486276.537216 punch from 0x024fa2 to 0x0279ec (0x2a4b bytes) 000650 1713486276.539219 punch from 0x0269b5 to 0x02d02c (0x6678 bytes) 000652 1713486276.540690 punch from 0x011b16 to 0x01fd0a (0xe1f5 bytes) 000655 1713486276.544349 fallocate 0x03885a thru 0x03ffff (0x77a6 bytes) 000657 1713486276.545828 fallocate 0x022970 thru 0x02c8bb (0x9f4c bytes) 000658 1713486276.548201 MIRROR_EXTEND to 12 mirrors 000660 1713486276.679505 fallocate 0x038ac9 thru 0x03ffff (0x7537 bytes) 000661 1713486276.693297 MIRROR_RESYNC 12 mirrors 000662 1713486277.099244 write 0x02bcef thru 0x033e81 (0x8193 bytes) 000663 1713486277.112653 fallocate 0x02ce41 thru 0x034da4 (0x7f64 bytes) 000664 1713486277.114146 mapwrite 0x01ada9 thru 0x0272ef (0xc547 bytes) 000666 1713486277.130018 read 0x0302aa thru 0x038e12 (0x8b69 bytes) 000667 1713486277.133997 fallocate 0x019a08 thru 0x029789 (0xfd82 bytes) 000668 1713486277.135867 trunc from 0x040000 to 0x01cc05 (0x233fa bytes) 000669 1713486277.149275 trunc from 0x01cc06 to 0x0241e2 (0x75dd bytes) 000670 1713486277.154294 mapread 0x019f7f thru 0x01a0f9 (0x17b bytes) 000671 1713486277.154977 fallocate 0x024f34 thru 0x02ad2a (0x5df7 bytes) 000672 1713486277.157086 write 0x018271 thru 0x01e124 (0x5eb4 bytes) 000673 1713486277.159095 trunc from 0x02ad2b to 0x02f7e6 (0x4abc bytes) 000674 1713486277.162907 write 0x0009e2 thru 0x010075 (0xf694 bytes) 000675 1713486277.167471 write 0x001e45 thru 0x00f6d1 (0xd88d bytes) 000677 1713486277.168359 read 0x0117fb thru 0x017aa5 (0x62ab bytes) 000679 1713486277.171325 fallocate 0x03eaf6 thru 0x03ffff (0x150a bytes) 000680 1713486277.172833 trunc from 0x02f7e7 to 0x01dfb2 (0x11834 bytes) 000681 1713486277.178025 MIRROR_RESYNC 12 mirrors 000682 1713486277.547977 mapwrite 0x00b020 thru 0x019347 (0xe328 bytes) 000684 1713486277.589033 read 0x00dfa1 thru 0x019132 (0xb192 bytes) 000686 1713486277.589735 mapread 0x01b61c thru 0x01dfb2 (0x2997 bytes) 000688 1713486277.590653 read 0x002255 thru 0x006421 (0x41cd bytes) 000690 1713486277.594056 mapread 0x0093dc thru 0x01163d (0x8262 bytes) 000691 1713486277.594660 mapwrite 0x0148db thru 0x015407 (0xb2d bytes) 000693 1713486277.600025 trunc from 0x01dfb3 to 0x0289ca (0xaa18 bytes) 000694 1713486277.614058 write 0x02aaf9 thru 0x031bec (0x70f4 bytes) 000695 1713486277.615605 write 0x003471 thru 0x009810 (0x63a0 bytes) 000696 1713486277.616396 fallocate 0x02272c thru 0x0263c8 (0x3c9d bytes) 000697 1713486277.619046 fallocate 0x01a003 thru 0x024639 (0xa637 bytes) 000698 1713486277.622390 MIRROR_EXTEND to 13 mirrors 000699 1713486277.758284 MIRROR_RESYNC 13 mirrors 000700 1713486278.023320 read 0x000a73 thru 0x009558 (0x8ae6 bytes) 000702 1713486278.026076 mapwrite 0x038d24 thru 0x03ffff (0x72dc bytes) 000703 1713486278.051292 mapread 0x000e6e thru 0x00a786 (0x9919 bytes) 000704 1713486278.057056 trunc from 0x040000 to 0x01639d (0x29c62 bytes) 000705 1713486278.062874 read 0x00e55c thru 0x0150dc (0x6b81 bytes) 000706 1713486278.065047 mapwrite 0x039253 thru 0x03ffff (0x6dad bytes) 000707 1713486278.078482 mapwrite 0x02a02a thru 0x0389a2 (0xe979 bytes) 000708 1713486278.086752 read 0x02e6d0 thru 0x03d2a3 (0xebd4 bytes) 000709 1713486278.087200 mapwrite 0x025805 thru 0x02c349 (0x6b45 bytes) 000710 1713486278.093058 mapread 0x03516a thru 0x03ccd1 (0x7b68 bytes) 000711 1713486278.093542 mapread 0x01ad86 thru 0x022558 (0x77d3 bytes) 000712 1713486278.094127 mapread 0x021297 thru 0x02526f (0x3fd9 bytes) 000714 1713486278.094905 read 0x036536 thru 0x03edb1 (0x887c bytes) 000715 1713486278.095307 mapwrite 0x02ba8c thru 0x032b8f (0x7104 bytes) 000717 1713486278.102337 read 0x02730f thru 0x02fe7c (0x8b6e bytes) 000719 1713486278.103132 write 0x039701 thru 0x03ce72 (0x3772 bytes) 000721 1713486278.104039 punch from 0x02ce49 to 0x02f294 (0x244c bytes) 000722 1713486278.106031 fallocate 0x004b2c thru 0x0092bb (0x4790 bytes) 000723 1713486278.108333 mapwrite 0x0293fd thru 0x02cf02 (0x3b06 bytes) 000724 1713486278.114810 punch from 0x00f311 to 0x01997c (0xa66c bytes) 000725 1713486278.116426 mapread 0x015d0d thru 0x024c6d (0xef61 bytes) 000726 1713486278.121606 read 0x02d2cc thru 0x034fcf (0x7d04 bytes) 000727 1713486278.123186 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000728 1713486278.167241 fallocate 0x03aee6 thru 0x03ffff (0x511a bytes) 000729 1713486278.169716 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000732 1713486278.218510 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000733 1713486278.262176 read 0x03c7a9 thru 0x03dae3 (0x133b bytes) 000734 1713486278.263814 write 0x01f910 thru 0x01fb37 (0x228 bytes) 000736 1713486278.267301 MIRROR_RESYNC 10 mirrors 000737 1713486278.617151 mapwrite 0x028cc4 thru 0x02c462 (0x379f bytes) 000738 1713486278.651736 MIRROR_RESYNC 10 mirrors 000739 1713486279.019208 read 0x026d39 thru 0x02f05e (0x8326 bytes) 000740 1713486279.022263 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000741 1713486279.093536 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000742 1713486279.166082 mapread 0x028942 thru 0x0309ee (0x80ad bytes) 000743 1713486279.170498 trunc from 0x040000 to 0x00ec96 (0x31369 bytes) 000744 1713486279.185762 mapwrite 0x03718a thru 0x03a542 (0x33b9 bytes) 000745 1713486279.195526 punch from 0x0229f8 to 0x0275cc (0x4bd5 bytes) 000746 1713486279.196984 punch from 0x0158bc to 0x016c5c (0x13a1 bytes) 000747 1713486279.198125 write 0x006f0c thru 0x00c895 (0x598a bytes) 000748 1713486279.200527 fallocate 0x015294 thru 0x02286c (0xd5d9 bytes) 000749 1713486279.202899 MIRROR_EXTEND to 9 mirrors 000750 1713486279.304321 write 0x02b522 thru 0x02de9c (0x297b bytes) 000752 1713486279.313501 write 0x01aa5b thru 0x02562d (0xabd3 bytes) 000753 1713486279.316863 mapwrite 0x00de1f thru 0x01293e (0x4b20 bytes) 000754 1713486279.324307 trunc from 0x03a543 to 0x017249 (0x232f9 bytes) 000755 1713486279.329292 write 0x02f4d1 thru 0x03be16 (0xc946 bytes) 000756 1713486279.330396 write 0x01a0ff thru 0x022003 (0x7f05 bytes) 000757 1713486279.332685 punch from 0x019f3d to 0x0203d4 (0x6498 bytes) 000758 1713486279.337072 fallocate 0x02a334 thru 0x02e80a (0x44d7 bytes) 000760 1713486279.338845 trunc from 0x03be17 to 0x02827e (0x13b98 bytes) 000761 1713486279.342361 trunc from 0x02827f to 0x0254f7 (0x2d87 bytes) 000762 1713486279.345560 write 0x012279 thru 0x0210c7 (0xee4f bytes) 000763 1713486279.346475 fallocate 0x005f95 thru 0x0082e2 (0x234e bytes) 000764 1713486279.348929 MIRROR_EXTEND to 10 mirrors 000765 1713486279.462353 read 0x00fedb thru 0x01ea00 (0xeb26 bytes) 000766 1713486279.467301 fallocate 0x02907a thru 0x03124c (0x81d3 bytes) 000767 1713486279.478686 read 0x011999 thru 0x01e762 (0xcdca bytes) 000768 1713486279.481677 trunc from 0x0254f8 to 0x039bbc (0x146c5 bytes) 000769 1713486279.487446 fallocate 0x002eba thru 0x01052b (0xd672 bytes) 000770 1713486279.490112 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000771 1713486279.533719 mapwrite 0x00ebfd thru 0x00f5eb (0x9ef bytes) 000772 1713486279.539878 punch from 0x0345a5 to 0x038f72 (0x49ce bytes) 000774 1713486279.541298 fallocate 0x028da0 thru 0x02efbb (0x621c bytes) 000775 1713486279.542557 punch from 0x014506 to 0x01f241 (0xad3c bytes) 000776 1713486279.544630 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000777 1713486279.587865 fallocate 0x01f902 thru 0x026f3f (0x763e bytes) 000778 1713486279.589359 read 0x00f211 thru 0x01b5a4 (0xc394 bytes) 000779 1713486279.591620 read 0x028948 thru 0x029b47 (0x1200 bytes) 000780 1713486279.594847 trunc from 0x039bbd to 0x00142b (0x38791 bytes) 000781 1713486279.600234 write 0x03e54e thru 0x03ffff (0x1ab2 bytes) 000783 1713486279.602405 MIRROR_EXTEND to 9 mirrors 000784 1713486279.714532 punch from 0x03f389 to 0x03ffff (0xc77 bytes) 000785 1713486279.725119 mapread 0x01f54c thru 0x0255e3 (0x6098 bytes) 000786 1713486279.729952 read 0x00030d thru 0x003fdb (0x3ccf bytes) 000787 1713486279.731823 fallocate 0x01f8c0 thru 0x02e963 (0xf0a4 bytes) 000788 1713486279.734763 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000789 1713486279.774247 punch from 0x032b63 to 0x03850d (0x59ab bytes) 000790 1713486279.775664 mapread 0x005203 thru 0x00cc1b (0x7a19 bytes) 000791 1713486279.779493 read 0x0204c7 thru 0x02143b (0xf75 bytes) 000792 1713486279.780902 mapread 0x0259b7 thru 0x02dfef (0x8639 bytes) 000793 1713486279.784725 punch from 0x03970c to 0x03995d (0x252 bytes) 000794 1713486279.787032 MIRROR_EXTEND to 9 mirrors 000796 1713486279.889780 punch from 0x01e683 to 0x0299e3 (0xb361 bytes) 000797 1713486279.898732 write 0x01086f thru 0x01f020 (0xe7b2 bytes) 000798 1713486279.902587 write 0x009c47 thru 0x0157fd (0xbbb7 bytes) 000799 1713486279.905363 trunc from 0x040000 to 0x01a370 (0x25c8f bytes) 000800 1713486279.913723 trunc from 0x01a371 to 0x02ae8c (0x10b1c bytes) 000801 1713486279.917421 mapread 0x00a9f0 thru 0x00de94 (0x34a5 bytes) 000802 1713486279.917957 mapwrite 0x0115e6 thru 0x0201ef (0xec0a bytes) 000803 1713486279.926848 punch from 0x001e5e to 0x00b89f (0x9a42 bytes) 000804 1713486279.928407 mapread 0x01fab4 thru 0x02ae8c (0xb3d9 bytes) 000805 1713486279.931076 read 0x0282a6 thru 0x02ae8c (0x2be7 bytes) 000806 1713486279.932375 MIRROR_RESYNC 9 mirrors 000807 1713486280.224373 write 0x012a23 thru 0x01bc64 (0x9242 bytes) 000808 1713486280.234499 fallocate 0x024609 thru 0x02c5d7 (0x7fcf bytes) 000809 1713486280.236119 mapwrite 0x036a10 thru 0x03bb87 (0x5178 bytes) 000811 1713486280.252658 write 0x02d2f4 thru 0x03bc2e (0xe93b bytes) 000812 1713486280.257721 MIRROR_EXTEND to 10 mirrors 000813 1713486280.394058 MIRROR_EXTEND to 11 mirrors 000814 1713486280.536776 mapread 0x0385d8 thru 0x03bc2e (0x3657 bytes) 000815 1713486280.542456 fallocate 0x025f94 thru 0x02b3e9 (0x5456 bytes) 000818 1713486280.555781 read 0x0202a9 thru 0x026cda (0x6a32 bytes) 000819 1713486280.558666 write 0x00002a thru 0x0074f9 (0x74d0 bytes) 000820 1713486280.562688 read 0x03588f thru 0x03a461 (0x4bd3 bytes) 000821 1713486280.565340 read 0x03ab10 thru 0x03bc2e (0x111f bytes) 000822 1713486280.566821 MIRROR_EXTEND to 12 mirrors 000823 1713486280.689287 read 0x00f5e5 thru 0x0177cd (0x81e9 bytes) 000824 1713486280.695110 MIRROR_EXTEND to 13 mirrors 000826 1713486280.849115 mapwrite 0x03bbc6 thru 0x03f581 (0x39bc bytes) 000827 1713486280.886033 trunc from 0x03f582 to 0x03ddfc (0x1785 bytes) 000828 1713486280.896341 fallocate 0x007f21 thru 0x00880c (0x8ec bytes) 000830 1713486280.899253 trunc from 0x03ddfd to 0x017242 (0x26bba bytes) 000831 1713486280.907203 trunc from 0x017243 to 0x03429c (0x1d05a bytes) 000832 1713486280.913274 trunc from 0x03429d to 0x02d88c (0x6a10 bytes) 000833 1713486280.917406 mapwrite 0x0072a4 thru 0x01124d (0x9faa bytes) 000835 1713486280.931511 punch from 0x01e94b to 0x02215e (0x3814 bytes) 000838 1713486280.936063 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000839 1713486280.993294 write 0x01eac6 thru 0x024451 (0x598c bytes) 000840 1713486280.999040 MIRROR_EXTEND to 13 mirrors 000841 1713486281.211809 mapread 0x01807a thru 0x021b0a (0x9a91 bytes) 000842 1713486281.225770 fallocate 0x00675e thru 0x00fa4d (0x92f0 bytes) 000843 1713486281.242212 punch from 0x02d305 to 0x02d88c (0x588 bytes) 000845 1713486281.244720 fallocate 0x024728 thru 0x029ce4 (0x55bd bytes) 000846 1713486281.248147 MIRROR_EXTEND to 14 mirrors 000847 1713486281.407725 MIRROR_RESYNC 14 mirrors 000848 1713486281.955215 punch from 0x01662b to 0x0198c6 (0x329c bytes) 000850 1713486281.968093 MIRROR_EXTEND to 15 mirrors 000851 1713486282.101350 fallocate 0x030a62 thru 0x030b3d (0xdc bytes) 000853 1713486282.113885 write 0x011ceb thru 0x0178f9 (0x5c0f bytes) 000855 1713486282.117908 mapwrite 0x03cfe0 thru 0x03d78c (0x7ad bytes) 000856 1713486282.130239 read 0x027055 thru 0x030669 (0x9615 bytes) 000857 1713486282.132994 mapread 0x03907b thru 0x03d78c (0x4712 bytes) 000858 1713486282.133573 mapwrite 0x000fb7 thru 0x008955 (0x799f bytes) 000859 1713486282.142482 mapread 0x02b28c thru 0x03158c (0x6301 bytes) 000860 1713486282.146229 punch from 0x006f69 to 0x009805 (0x289d bytes) 000861 1713486282.148315 mapread 0x002799 thru 0x00c2de (0x9b46 bytes) 000862 1713486282.151638 mapread 0x00cce3 thru 0x012ef7 (0x6215 bytes) 000863 1713486282.152175 trunc from 0x03d78d to 0x003b8e (0x39bfe bytes) 000865 1713486282.157877 mapread 0x0009e7 thru 0x003b8e (0x31a8 bytes) 000866 1713486282.159950 MIRROR_SPLIT mirror 12 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000867 1713486282.217882 fallocate 0x0216a5 thru 0x02e06b (0xc9c7 bytes) 000868 1713486282.220187 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000869 1713486282.272141 fallocate 0x012a6b thru 0x01eff5 (0xc58b bytes) 000870 1713486282.274065 mapread 0x0006c4 thru 0x003b8e (0x34cb bytes) 000871 1713486282.281580 MIRROR_RESYNC 13 mirrors 000872 1713486282.477999 MIRROR_EXTEND to 14 mirrors 000873 1713486282.585538 mapwrite 0x02f03c thru 0x030c35 (0x1bfa bytes) 000874 1713486282.611319 fallocate 0x01a333 thru 0x029dc5 (0xfa93 bytes) 000875 1713486282.613191 read 0x00cfa5 thru 0x01375f (0x67bb bytes) 000876 1713486282.615888 punch from 0x001a9d to 0x00c317 (0xa87b bytes) 000877 1713486282.617638 mapwrite 0x0263f2 thru 0x0320a6 (0xbcb5 bytes) 000878 1713486282.636158 fallocate 0x0282b8 thru 0x0344a2 (0xc1eb bytes) 000880 1713486282.638567 mapread 0x017556 thru 0x01cf3d (0x59e8 bytes) 000881 1713486282.642790 trunc from 0x0344a3 to 0x00ce28 (0x2767a bytes) 000882 1713486282.649878 MIRROR_RESYNC 14 mirrors 000883 1713486282.980564 fallocate 0x02771a thru 0x02c885 (0x516c bytes) 000884 1713486282.993751 read 0x00121c thru 0x00ce28 (0xbc0d bytes) 000885 1713486282.998820 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000887 1713486283.060631 read 0x0017b2 thru 0x00bd77 (0xa5c6 bytes) 000888 1713486283.065048 MIRROR_RESYNC 13 mirrors 000889 1713486283.363241 mapread 0x009c56 thru 0x00ada6 (0x1151 bytes) 000890 1713486283.365414 read 0x001bea thru 0x0037c7 (0x1bde bytes) 000892 1713486283.366696 trunc from 0x00ce29 to 0x005676 (0x77b2 bytes) 000893 1713486283.376361 punch from 0x002aaf to 0x005676 (0x2bc8 bytes) 000895 1713486283.379315 MIRROR_EXTEND to 14 mirrors 000896 1713486283.533318 mapread 0x000fed thru 0x005676 (0x468a bytes) 000898 1713486283.545885 mapwrite 0x01db8d thru 0x02d3ca (0xf83e bytes) 000899 1713486283.581232 fallocate 0x035bc7 thru 0x036379 (0x7b3 bytes) 000901 1713486283.583131 mapread 0x023331 thru 0x02d3ca (0xa09a bytes) 000902 1713486283.583638 write 0x03dcc2 thru 0x03ffff (0x233e bytes) 000903 1713486283.584314 trunc from 0x040000 to 0x02b3ce (0x14c31 bytes) 000904 1713486283.591803 mapread 0x020a1c thru 0x02b3ce (0xa9b3 bytes) 000906 1713486283.592795 read 0x023edd thru 0x02b3ce (0x74f2 bytes) 000907 1713486283.593240 mapread 0x01eda1 thru 0x02b3ce (0xc62e bytes) 000909 1713486283.594158 write 0x03dc28 thru 0x03ffff (0x23d8 bytes) 000910 1713486283.594957 mapread 0x004290 thru 0x00a57c (0x62ed bytes) 000911 1713486283.599679 fallocate 0x03cb0f thru 0x03f769 (0x2c5b bytes) 000913 1713486283.602180 punch from 0x02f3a7 to 0x0375db (0x8235 bytes) 000914 1713486283.604418 punch from 0x0387bc to 0x03ffff (0x7844 bytes) 000915 1713486283.610294 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000916 1713486283.680540 mapread 0x0341c0 thru 0x037c4b (0x3a8c bytes) 000917 1713486283.685305 read 0x02c957 thru 0x03501e (0x86c8 bytes) 000918 1713486283.686971 trunc from 0x040000 to 0x002273 (0x3dd8c bytes) 000919 1713486283.691814 fallocate 0x00a243 thru 0x00dc35 (0x39f3 bytes) 000920 1713486283.693648 mapread 0x0015ea thru 0x00922b (0x7c42 bytes) 000921 1713486283.698619 MIRROR_RESYNC 13 mirrors 000923 1713486284.106630 trunc from 0x00dc36 to 0x02309b (0x15466 bytes) 000924 1713486284.126710 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000925 1713486284.210194 read 0x0091ee thru 0x00a952 (0x1765 bytes) 000926 1713486284.212243 fallocate 0x0079ba thru 0x00e33f (0x6986 bytes) 000927 1713486284.214134 read 0x00e892 thru 0x012eb1 (0x4620 bytes) 000928 1713486284.218390 read 0x0095bc thru 0x011151 (0x7b96 bytes) 000929 1713486284.218841 fallocate 0x034c32 thru 0x035c9b (0x106a bytes) 000930 1713486284.220697 trunc from 0x02309c to 0x00eaa2 (0x145f9 bytes) 000931 1713486284.226967 fallocate 0x02cf84 thru 0x03ccd6 (0xfd53 bytes) 000932 1713486284.228844 mapread 0x002fbe thru 0x00b9fa (0x8a3d bytes) 000934 1713486284.231256 MIRROR_RESYNC 12 mirrors 000936 1713486284.657976 read 0x00dc4e thru 0x00eaa2 (0xe55 bytes) 000938 1713486284.661636 MIRROR_EXTEND to 13 mirrors 000939 1713486284.822223 write 0x01d484 thru 0x01ef8a (0x1b07 bytes) 000940 1713486284.836303 read 0x01e18d thru 0x01ef8a (0xdfe bytes) 000943 1713486284.837373 read 0x00ae28 thru 0x00c40b (0x15e4 bytes) 000945 1713486284.844441 MIRROR_EXTEND to 14 mirrors 000946 1713486285.001835 fallocate 0x013799 thru 0x019b4d (0x63b5 bytes) 000947 1713486285.013429 mapwrite 0x0213c7 thru 0x02b444 (0xa07e bytes) 000948 1713486285.039992 punch from 0x018e6e to 0x01b0fb (0x228e bytes) 000949 1713486285.042066 mapwrite 0x017442 thru 0x01905c (0x1c1b bytes) 000950 1713486285.051262 mapwrite 0x0138d1 thru 0x01e802 (0xaf32 bytes) 000951 1713486285.060369 mapwrite 0x029798 thru 0x03054d (0x6db6 bytes) 000952 1713486285.075637 fallocate 0x008a77 thru 0x01471e (0xbca8 bytes) 000953 1713486285.077531 punch from 0x0039f9 to 0x00f645 (0xbc4d bytes) 000956 1713486285.079710 read 0x0193a5 thru 0x019508 (0x164 bytes) 000957 1713486285.080078 write 0x0307c6 thru 0x0320e1 (0x191c bytes) 000958 1713486285.080717 read 0x00e82b thru 0x013e08 (0x55de bytes) 000959 1713486285.085003 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000960 1713486285.142870 write 0x030a25 thru 0x0390f5 (0x86d1 bytes) 000961 1713486285.146544 write 0x01b368 thru 0x01bb83 (0x81c bytes) 000962 1713486285.148287 mapread 0x01c960 thru 0x023523 (0x6bc4 bytes) 000963 1713486285.150981 trunc from 0x0390f6 to 0x0205bf (0x18b36 bytes) 000964 1713486285.155156 mapwrite 0x024785 thru 0x0326ce (0xdf4a bytes) 000966 1713486285.173240 read 0x014a79 thru 0x016bb8 (0x2140 bytes) 000968 1713486285.175349 fallocate 0x009648 thru 0x0161e8 (0xcba1 bytes) 000970 1713486285.177169 fallocate 0x03b898 thru 0x03ffff (0x4768 bytes) 000971 1713486285.178889 punch from 0x01ecaf to 0x027880 (0x8bd2 bytes) 000973 1713486285.181235 read 0x004337 thru 0x0060aa (0x1d74 bytes) 000974 1713486285.185247 mapread 0x00586f thru 0x00cdd6 (0x7568 bytes) 000976 1713486285.186198 mapread 0x0341f7 thru 0x03ffff (0xbe09 bytes) 000977 1713486285.186682 read 0x037009 thru 0x037481 (0x479 bytes) 000978 1713486285.187061 punch from 0x02e1a9 to 0x03956c (0xb3c4 bytes) 000979 1713486285.189164 fallocate 0x0191b8 thru 0x0259dd (0xc826 bytes) 000980 1713486285.190750 write 0x016a5d thru 0x023831 (0xcdd5 bytes) 000981 1713486285.191605 mapwrite 0x010bc8 thru 0x012c96 (0x20cf bytes) 000982 1713486285.197513 write 0x00263e thru 0x007657 (0x501a bytes) 000985 1713486285.198703 fallocate 0x02765b thru 0x036540 (0xeee6 bytes) 000986 1713486285.200628 mapread 0x031c35 thru 0x034500 (0x28cc bytes) 000988 1713486285.204324 read 0x03c58d thru 0x03ffff (0x3a73 bytes) 000989 1713486285.204722 fallocate 0x03f624 thru 0x03feb9 (0x896 bytes) 000990 1713486285.206700 fallocate 0x00a89f thru 0x01042d (0x5b8f bytes) 000991 1713486285.208365 fallocate 0x01aaad thru 0x02394f (0x8ea3 bytes) 000992 1713486285.210168 trunc from 0x040000 to 0x02ec8b (0x11374 bytes) 000993 1713486285.214428 mapread 0x003c6d thru 0x013905 (0xfc99 bytes) 000994 1713486285.215146 punch from 0x01dee3 to 0x02ca88 (0xeba6 bytes) 000996 1713486285.219729 MIRROR_EXTEND to 14 mirrors 000997 1713486285.381088 mapwrite 0x034d00 thru 0x03ffff (0xb300 bytes) 000998 1713486285.417610 trunc from 0x040000 to 0x01262b (0x2d9d4 bytes) 000999 1713486285.422981 write 0x02812a thru 0x0297d8 (0x16af bytes) 001000 1713486285.425104 MIRROR_RESYNC 14 mirrors All operations completed A-OK! PASS 70a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 20:24:48 (1713486288) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre2 Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7225888 .. Extending file size to 4188000 .. done Extending file size to 8553280 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7821504 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8501248 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9081632 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8754368 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8941120 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 9521 9522 9523 9525 9527 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) 192.168.201.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:25:54 (1713486354) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre2 Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1759456 .. Extending file size to 3462880 .. done Extending file size to 6044768 .. Extending file size to 2890304 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7392800 .. Extending file size to 6535040 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7926336 .. Extending file size to 7521376 .. done Extending file size to 8965056 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8579680 .. failed Extending file size to 9224448 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9165856 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9336896 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 12814 12815 12817 12818 12821 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) 192.168.201.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:27:01 (1713486421) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3348 1284340 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2300 1285388 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14892 3592128 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23040 3583980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37932 7176108 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3348 1284340 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2300 1285388 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14892 3592128 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23040 3583980 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37932 7176108 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0363909 s, 57.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0495388 s, 42.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:27:08 (1713486428) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0338603 s, 61.9 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 20:27:11 (1713486431) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe6:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe8:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:27:15 (1713486435) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0260006 s, 80.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:27:19 (1713486439) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0317022 s, 66.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbea:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:27:22 (1713486442) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0276557 s, 75.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbeb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbeb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbec:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:27:47 (1713486467) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156138 s, 67.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbea:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbea:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbeb:0x0] } PASS 204d (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:28:18 (1713486498) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0274909 s, 76.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:28:53 (1713486533) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg119-server: Pool lustre.test_204f created Adding targets to pool oleg119-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg119-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0175723 s, 119 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg119-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg119-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg119-server: Pool lustre.test_204f destroyed PASS 204f (45s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 20:29:40 (1713486580) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:29:43 (1713486583) oleg119-server: Pool lustre.test_206 created Adding targets to pool oleg119-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg119-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg119-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg119-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg119-server: Pool lustre.test_206 destroyed PASS 206 (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:29:58 (1713486598) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186974 s, 56.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100593 s, 104 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x69d509e7 Mirror 3: 0x69d509e7 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:30:02 (1713486602) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:30:05 (1713486605) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:30:07 (1713486607) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00923088 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x19b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ef0000 Read this (4 bytes): BBBB 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:30:22 (1713486622) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00280823 s, 2.9 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 20:30:25 (1713486625) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111115 s, 94.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x65f lmm_fid: [0x200000405:0x65f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x65f lmm_fid: [0x200000405:0x65f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf8:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612767 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-18 20:30:26.000000000 -0400 Modify: 2024-04-18 20:30:26.000000000 -0400 Change: 2024-04-18 20:30:26.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:30:29 (1713486629) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00946384 s, 111 MB/s oleg119-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg119-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg119-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg119-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg119-server: O/280000401/d23/bf7: File not found by ext2_lookup PASS 210b (9s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 839 sec ============ 20:30:39 (1713486639) === sanity-flr: start cleanup 20:30:39 (1713486639) === === sanity-flr: finish cleanup 20:30:40 (1713486640) ===