-----============= acceptance-small: sanity-flr ============----- Tue Apr 16 10:49:51 EDT 2024 excepting tests: 6 201 44c === sanity-flr: start setup 10:49:54 (1713278994) === oleg259-client.virtnet: executing check_config_client /mnt/lustre oleg259-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg259-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b3503000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b3503000.idle_timeout=debug disable quota as required oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:50:01 (1713279001) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7503] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 10:50:02 (1713279002) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:50:07 (1713279007) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:50:09 (1713279009) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 10:50:11 (1713279011) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:50:17 (1713279017) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:50:19 (1713279019) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 10:50:21 (1713279021) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b3503000.stats=clear osc.lustre-OST0001-osc-ffff8800b3503000.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:50:25 (1713279025) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:50:28 (1713279028) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:50:31 (1713279031) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 10:50:34 (1713279034) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 10:50:37 (1713279037) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:50:40 (1713279040) PASS 3 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:50:43 (1713279043) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:50:46 (1713279046) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 10:50:50 (1713279050) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:50:53 (1713279053) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.458224 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.028184 s, 37.2 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:50:57 (1713279057) 13+0 records in 13+0 records out 13631488 bytes (14 MB) copied, 0.0943109 s, 145 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:51:01 (1713279061) Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3503000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3503000.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3503000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3503000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:51:23 (1713279083) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.101864 s, 61.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3503000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3503000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:51:33 (1713279093) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3503000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3503000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3503000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3503000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:51:52 (1713279112) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.834165 s, 62.9 MB/s -rw-r--r-- 1 root root 52428800 Apr 16 10:51 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.200s user 0m0.000s sys 0m0.135s Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3503000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3503000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.194s user 0m0.001s sys 0m0.121s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3503000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3503000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.205s user 0m0.001s sys 0m0.119s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b3503000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3503000.ost_server_uuid in FULL state after 0 sec PASS 33b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:52:13 (1713279133) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:52:15 (1713279135) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:52:17 (1713279137) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 10:52:20 (1713279140) PASS 35 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 10:52:23 (1713279143) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.123326 s, 102 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:52:27 (1713279147) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00965787 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00893048 s, 117 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:52:30 (1713279150) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:52:34 (1713279154) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000523158 s, 2.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000558068 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000511161 s, 2.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000408914 s, 2.5 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 10:52:38 (1713279158) create test files with size 15759878 .. 68fb39f97722c4dec423d486f08b1ac2 - 61fff7d3b2dc8f641d645853c1dec1aa - deed523c6fd03f8930a1613a356bfdc0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 10:52:44 (1713279164) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:52:48 (1713279168) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:52:51 (1713279171) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149548 s, 70.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00924805 s, 113 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 10:52:55 (1713279175) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0189055 s, 111 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0871846 s, 48.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0343878 s, 61.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 10:52:58 (1713279178) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 10:53:01 (1713279181) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:53:03 (1713279183) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118499 s, 88.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 10:53:06 (1713279186) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0678918 s, 46.3 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:53:10 (1713279190) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:53:13 (1713279193) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0737166 s, 142 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:53:17 (1713279197) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800aab01000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0700253 s, 150 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a79a6800.xattr_cache=1 PASS 44e (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:53:22 (1713279202) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 10:53:30 (1713279210) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 10:53:37 (1713279217) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 10:53:39 (1713279219) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0624559 s, 50.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0478419 s, 65.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 735797b79253558236140856a31b94cd - mirror 1 checksum: b6ad7fd4648bc0fe64a085a001bef020 - mirror 2 checksum: 735797b79253558236140856a31b94cd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 10:53:40 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 10:53:40 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 735797b79253558236140856a31b94cd - mirror 1 checksum: b6ad7fd4648bc0fe64a085a001bef020 - mirror 2 checksum: b6ad7fd4648bc0fe64a085a001bef020 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 10:53:40 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 10:53:40 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 48 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:53:43 (1713279223) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00952559 s, 110 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:53:46 (1713279226) Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 10:53:53 (1713279233) shut down Failover mds1 to oleg259-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 10:54:06 (1713279246) targets are mounted 10:54:06 (1713279246) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00803086 s, 131 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 50A (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:54:15 (1713279255) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00714956 s, 2.9 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00398971 s, 5.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 10:54:18 (1713279258) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00856059 s, 2.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0033623 s, 6.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:54:22 (1713279262) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00237113 s, 6.9 MB/s PASS 50c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 10:54:25 (1713279265) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.010003 s, 2.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:54:28 (1713279268) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00146243 s, 2.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00712502 s, 575 kB/s PASS 60a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:54:31 (1713279271) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:54:37 (1713279277) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:54:56 (1713279296) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:55:14 (1713279314) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:55:32 (1713279332) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 10:55:35 (1713279335) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 30461 mirror split pid 30462 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7301: 30462 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 30461 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 10:56:39 (1713279399) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 9399 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero length punch hole truncating to largest ever: 0x2055 000004 1713279399.531287 trunc from 00000000 to 0x002054 (0x2055 bytes) truncating to largest ever: 0x267bb 000006 1713279399.535555 trunc from 0x002055 to 0x0267ba (0x24766 bytes) 000007 1713279399.539026 mapwrite 0x01b352 thru 0x026d6d (0xba1c bytes) 000008 1713279399.555416 mapread 0x020bdc thru 0x0256b0 (0x4ad5 bytes) 000009 1713279399.555869 read 0x00ecef thru 0x01af81 (0xc293 bytes) fallocating to largest ever: 0x2ab7f 000010 1713279399.558224 fallocate 0x01c635 thru 0x02ab7e (0xe54a bytes) fallocating to largest ever: 0x314a3 000012 1713279399.560453 fallocate 0x023304 thru 0x0314a2 (0xe19f bytes) 000014 1713279399.562468 punch from 0x00bc8e to 0x012ed4 (0x7247 bytes) 000015 1713279399.563814 read 0x02fbc0 thru 0x0314a2 (0x18e3 bytes) truncating to largest ever: 0x316bf 000016 1713279399.567105 trunc from 0x0314a3 to 0x0316be (0x21c bytes) 000017 1713279399.571202 punch from 0x02c739 to 0x0316be (0x4f86 bytes) 000019 1713279399.574155 MIRROR_EXTEND to 2 mirrors 000020 1713279399.676814 MIRROR_RESYNC 2 mirrors 000021 1713279399.706707 write 0x01e350 thru 0x0232b3 (0x4f64 bytes) 000024 1713279399.716531 mapwrite 0x01acd7 thru 0x02a19b (0xf4c5 bytes) 000026 1713279399.727258 write 0x032690 thru 0x03e39d (0xbd0e bytes) 000028 1713279399.728185 mapread 0x00f9e5 thru 0x0145b0 (0x4bcc bytes) 000030 1713279399.730438 fallocate 0x0020f6 thru 0x004e16 (0x2d21 bytes) 000032 1713279399.731999 punch from 0x03c66b to 0x03e39d (0x1d33 bytes) fallocating to largest ever: 0x40000 000033 1713279399.734923 fallocate 0x034635 thru 0x03ffff (0xb9cb bytes) 000034 1713279399.736091 trunc from 0x040000 to 0x033a20 (0xc5df bytes) 000035 1713279399.744633 fallocate 0x01415c thru 0x01c9f0 (0x8895 bytes) 000036 1713279399.745947 fallocate 0x01f3c7 thru 0x02cbff (0xd839 bytes) 000037 1713279399.747376 fallocate 0x02626b thru 0x035397 (0xf12d bytes) 000039 1713279399.748960 write 0x025b74 thru 0x02b5e6 (0x5a73 bytes) 000040 1713279399.749699 fallocate 0x01b0ed thru 0x028fc0 (0xded4 bytes) 000041 1713279399.751046 mapread 0x007e9f thru 0x010c14 (0x8d76 bytes) 000042 1713279399.753614 punch from 0x0220f3 to 0x031e84 (0xfd92 bytes) 000043 1713279399.756837 read 0x00d663 thru 0x016acb (0x9469 bytes) 000044 1713279399.758286 MIRROR_EXTEND to 3 mirrors 000046 1713279399.854977 write 0x01a10c thru 0x02580a (0xb6ff bytes) 000047 1713279399.867670 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000048 1713279399.912272 read 0x003084 thru 0x003b14 (0xa91 bytes) 000049 1713279399.913839 fallocate 0x033b65 thru 0x03ffff (0xc49b bytes) 000050 1713279399.915459 punch from 0x01f269 to 0x024b1e (0x58b6 bytes) 000051 1713279399.916823 read 0x01e4e3 thru 0x02994f (0xb46d bytes) 000053 1713279399.921417 MIRROR_EXTEND to 3 mirrors 000055 1713279400.023939 punch from 0x031010 to 0x033a20 (0x2a11 bytes) 000056 1713279400.032041 mapwrite 0x0268cf thru 0x034cec (0xe41e bytes) 000057 1713279400.052088 read 0x0254a8 thru 0x02a14b (0x4ca4 bytes) 000060 1713279400.052882 write 0x007303 thru 0x00f5c5 (0x82c3 bytes) 000061 1713279400.055440 mapread 0x00c423 thru 0x01afff (0xebdd bytes) 000063 1713279400.058758 read 0x00a172 thru 0x015b6b (0xb9fa bytes) 000064 1713279400.059024 trunc from 0x034ced to 0x002d2f (0x31fbd bytes) 000065 1713279400.063110 mapread 0x002715 thru 0x002d2f (0x61b bytes) 000066 1713279400.063512 trunc from 0x002d30 to 0x018d98 (0x16069 bytes) 000067 1713279400.066832 mapwrite 0x01a605 thru 0x0223d4 (0x7dd0 bytes) 000068 1713279400.079295 MIRROR_RESYNC 3 mirrors 000069 1713279400.180264 punch from 0x01a77c to 0x02022b (0x5ab0 bytes) 000071 1713279400.187746 read 0x01854f thru 0x0223d4 (0x9e86 bytes) 000072 1713279400.190723 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000073 1713279400.230956 punch from 0x020856 to 0x0223d4 (0x1b7f bytes) 000074 1713279400.233692 MIRROR_EXTEND to 3 mirrors 000075 1713279400.335414 trunc from 0x0223d5 to 0x028c66 (0x6892 bytes) 000076 1713279400.348407 mapwrite 0x031a7b thru 0x0326a7 (0xc2d bytes) 000077 1713279400.358601 fallocate 0x0239d3 thru 0x02a201 (0x682f bytes) 000078 1713279400.359961 read 0x02827f thru 0x0326a7 (0xa429 bytes) 000080 1713279400.362298 trunc from 0x0326a8 to 0x032eff (0x858 bytes) 000081 1713279400.366016 trunc from 0x032f00 to 0x015e65 (0x1d09a bytes) 000082 1713279400.369346 punch from 0x0130f0 to 0x015e65 (0x2d76 bytes) 000083 1713279400.370724 trunc from 0x015e66 to 0x027b52 (0x11ced bytes) 000085 1713279400.375113 MIRROR_EXTEND to 4 mirrors 000086 1713279400.475086 write 0x0164b7 thru 0x018365 (0x1eaf bytes) 000087 1713279400.486975 MIRROR_EXTEND to 5 mirrors 000088 1713279400.588461 punch from 0x003405 to 0x005207 (0x1e03 bytes) 000090 1713279400.596509 write 0x02b516 thru 0x03310d (0x7bf8 bytes) 000091 1713279400.598107 trunc from 0x03310e to 0x0042e8 (0x2ee25 bytes) 000092 1713279400.606490 MIRROR_RESYNC 5 mirrors 000093 1713279400.742770 punch from 0x003a57 to 0x0042e8 (0x892 bytes) 000094 1713279400.750245 write 0x004f76 thru 0x008723 (0x37ae bytes) 000095 1713279400.752613 MIRROR_RESYNC 5 mirrors 000096 1713279400.889988 trunc from 0x008724 to 0x00b6d0 (0x2fad bytes) 000097 1713279400.901879 trunc from 0x00b6d1 to 0x0219ec (0x1631c bytes) 000098 1713279400.905831 punch from 0x00349a to 0x00e259 (0xadc0 bytes) 000099 1713279400.907092 trunc from 0x0219ed to 0x023f8e (0x25a2 bytes) 000101 1713279400.910693 mapwrite 0x0250b3 thru 0x02ad05 (0x5c53 bytes) 000102 1713279400.922543 write 0x013283 thru 0x01814c (0x4eca bytes) 000103 1713279400.925180 mapwrite 0x00d0c0 thru 0x00e0dc (0x101d bytes) 000104 1713279400.929297 write 0x0080d8 thru 0x00993a (0x1863 bytes) 000105 1713279400.931443 fallocate 0x03ea64 thru 0x03ffff (0x159c bytes) 000106 1713279400.932761 fallocate 0x00f8e1 thru 0x01c651 (0xcd71 bytes) 000107 1713279400.934079 read 0x01de5c thru 0x024b6a (0x6d0f bytes) 000108 1713279400.935682 mapwrite 0x01962a thru 0x01b8aa (0x2281 bytes) 000109 1713279400.939902 mapwrite 0x00382b thru 0x00599a (0x2170 bytes) 000110 1713279400.943493 punch from 0x00fc99 to 0x01e45d (0xe7c5 bytes) 000111 1713279400.945032 fallocate 0x00a117 thru 0x00d158 (0x3042 bytes) 000112 1713279400.946449 punch from 0x011bd0 to 0x01d5f1 (0xba22 bytes) 000113 1713279400.947719 fallocate 0x03c408 thru 0x03ffff (0x3bf8 bytes) 000115 1713279400.950612 MIRROR_EXTEND to 6 mirrors 000116 1713279401.059028 fallocate 0x0085f8 thru 0x00e2d2 (0x5cdb bytes) 000118 1713279401.068947 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000119 1713279401.111884 mapread 0x000c77 thru 0x00c196 (0xb520 bytes) 000120 1713279401.117928 read 0x027ad0 thru 0x02ad05 (0x3236 bytes) 000121 1713279401.119296 trunc from 0x02ad06 to 0x03a90d (0xfc08 bytes) 000122 1713279401.124470 MIRROR_EXTEND to 6 mirrors 000123 1713279401.229289 trunc from 0x03a90e to 0x014e06 (0x25b07 bytes) 000124 1713279401.243295 MIRROR_EXTEND to 7 mirrors 000125 1713279401.348383 read 0x00e92e thru 0x014e06 (0x64d9 bytes) 000126 1713279401.352040 mapread 0x00cf7e thru 0x014e06 (0x7e89 bytes) 000127 1713279401.357585 mapwrite 0x035c97 thru 0x03ffff (0xa369 bytes) 000129 1713279401.382159 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000131 1713279401.431388 mapread 0x00d70a thru 0x015b12 (0x8409 bytes) 000135 1713279401.438011 mapread 0x014b88 thru 0x01f9bf (0xae38 bytes) 000139 1713279401.441504 read 0x01276e thru 0x021902 (0xf195 bytes) 000142 1713279401.442406 punch from 0x00c14e to 0x011158 (0x500b bytes) 000143 1713279401.443871 read 0x035924 thru 0x03a1eb (0x48c8 bytes) 000144 1713279401.446780 write 0x02494b thru 0x02f787 (0xae3d bytes) 000145 1713279401.450648 MIRROR_RESYNC 6 mirrors 000146 1713279401.699082 fallocate 0x03131c thru 0x03ef0e (0xdbf3 bytes) 000147 1713279401.709166 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000148 1713279401.753940 mapwrite 0x0001c6 thru 0x00e9eb (0xe826 bytes) 000150 1713279401.766077 read 0x00b492 thru 0x00e15b (0x2cca bytes) 000151 1713279401.766407 read 0x01fca9 thru 0x024889 (0x4be1 bytes) 000152 1713279401.771522 mapread 0x00b936 thru 0x01050c (0x4bd7 bytes) 000153 1713279401.772046 fallocate 0x014a49 thru 0x017b1c (0x30d4 bytes) 000158 1713279401.774898 write 0x03786c thru 0x03bed5 (0x466a bytes) 000159 1713279401.776519 mapread 0x0222c2 thru 0x02e8b1 (0xc5f0 bytes) 000160 1713279401.777107 punch from 0x001dbb to 0x00e83e (0xca84 bytes) 000161 1713279401.778714 write 0x03130f thru 0x037c3d (0x692f bytes) 000163 1713279401.780713 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000164 1713279401.823275 read 0x022e50 thru 0x032a54 (0xfc05 bytes) 000165 1713279401.825783 trunc from 0x040000 to 0x0094a9 (0x36b56 bytes) 000166 1713279401.833840 mapwrite 0x007bbc thru 0x013af6 (0xbf3b bytes) 000167 1713279401.848631 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000169 1713279401.889261 mapwrite 0x03392d thru 0x03e9e9 (0xb0bd bytes) 000170 1713279401.905093 read 0x02ac12 thru 0x02e48d (0x387c bytes) 000171 1713279401.906842 mapread 0x025b3b thru 0x02cd25 (0x71eb bytes) 000172 1713279401.909210 punch from 0x033c80 to 0x03e9e9 (0xad6a bytes) 000173 1713279401.910861 trunc from 0x03e9ea to 0x00dfa0 (0x30a49 bytes) 000174 1713279401.915979 MIRROR_EXTEND to 4 mirrors 000178 1713279402.016524 trunc from 0x00dfa1 to 0x016f63 (0x8fc3 bytes) 000179 1713279402.030099 write 0x00c582 thru 0x01292c (0x63ab bytes) 000180 1713279402.032795 punch from 0x008e8a to 0x016f63 (0xe0da bytes) 000181 1713279402.035984 punch from 0x011e89 to 0x016f63 (0x50db bytes) 000184 1713279402.039108 MIRROR_RESYNC 4 mirrors 000185 1713279402.176204 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000186 1713279402.225348 mapread 0x00c1cd thru 0x016f63 (0xad97 bytes) 000187 1713279402.231270 write 0x02dba1 thru 0x037659 (0x9ab9 bytes) 000188 1713279402.239955 trunc from 0x03765a to 0x00afdb (0x2c67e bytes) 000190 1713279402.244792 trunc from 0x00afdc to 0x002b74 (0x8467 bytes) 000191 1713279402.248276 punch from 0x001f06 to 0x002b74 (0xc6f bytes) 000192 1713279402.249595 mapread 0x0003b5 thru 0x002b74 (0x27c0 bytes) 000193 1713279402.252567 mapread 0x000026 thru 0x002b74 (0x2b4f bytes) 000194 1713279402.252949 read 0x001efe thru 0x002b74 (0xc77 bytes) 000196 1713279402.253470 punch from 0x001df5 to 0x002b74 (0xd80 bytes) 000197 1713279402.254732 write 0x02c705 thru 0x02c8a6 (0x1a2 bytes) 000198 1713279402.255306 punch from 0x00603e to 0x014db6 (0xed79 bytes) 000199 1713279402.256671 read 0x00f4b5 thru 0x0118c0 (0x240c bytes) 000200 1713279402.259291 write 0x03e653 thru 0x03ffff (0x19ad bytes) 000201 1713279402.259892 fallocate 0x0098a0 thru 0x00c3b7 (0x2b18 bytes) 000202 1713279402.261425 mapwrite 0x03278c thru 0x03ffff (0xd874 bytes) 000203 1713279402.269739 fallocate 0x032ccf thru 0x03985f (0x6b91 bytes) 000204 1713279402.271139 fallocate 0x0132b2 thru 0x01fbe9 (0xc938 bytes) 000205 1713279402.272603 mapwrite 0x02ece1 thru 0x036754 (0x7a74 bytes) 000206 1713279402.278857 MIRROR_RESYNC 3 mirrors 000207 1713279402.389699 trunc from 0x040000 to 0x011078 (0x2ef87 bytes) 000208 1713279402.401996 write 0x036070 thru 0x03fce9 (0x9c7a bytes) 000209 1713279402.403005 fallocate 0x01ed3e thru 0x01f937 (0xbfa bytes) 000210 1713279402.404618 write 0x01e025 thru 0x02d9f3 (0xf9cf bytes) 000211 1713279402.407467 mapwrite 0x001461 thru 0x009eb6 (0x8a56 bytes) 000213 1713279402.415777 punch from 0x0361e3 to 0x03fce9 (0x9b07 bytes) 000215 1713279402.417454 write 0x00cd86 thru 0x01266f (0x58ea bytes) 000217 1713279402.419383 read 0x0034a9 thru 0x00d225 (0x9d7d bytes) 000219 1713279402.419893 fallocate 0x015461 thru 0x020a9a (0xb63a bytes) 000220 1713279402.421058 mapwrite 0x011f32 thru 0x018f28 (0x6ff7 bytes) 000221 1713279402.427284 fallocate 0x02d8c3 thru 0x03a04b (0xc789 bytes) 000222 1713279402.429694 MIRROR_EXTEND to 4 mirrors 000223 1713279402.554760 punch from 0x0065f2 to 0x011fa8 (0xb9b7 bytes) 000224 1713279402.564917 trunc from 0x03fcea to 0x034415 (0xb8d4 bytes) 000226 1713279402.577487 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000227 1713279402.624405 punch from 0x0009fb to 0x001ae3 (0x10e9 bytes) 000228 1713279402.625993 trunc from 0x034416 to 0x034458 (0x43 bytes) 000229 1713279402.630532 mapread 0x0270ea thru 0x034458 (0xd36f bytes) 000230 1713279402.636808 punch from 0x0039c2 to 0x00e58a (0xabc9 bytes) 000231 1713279402.638255 mapread 0x0166e5 thru 0x022b0d (0xc429 bytes) 000232 1713279402.642521 fallocate 0x01d1c3 thru 0x023bb8 (0x69f6 bytes) 000234 1713279402.644079 trunc from 0x034459 to 0x03beb1 (0x7a59 bytes) 000235 1713279402.647768 mapread 0x0268b6 thru 0x03556c (0xecb7 bytes) 000236 1713279402.651310 MIRROR_EXTEND to 4 mirrors 000237 1713279402.760617 write 0x032ea0 thru 0x03ffff (0xd160 bytes) 000238 1713279402.769954 punch from 0x02b24d to 0x02f1c5 (0x3f79 bytes) 000239 1713279402.771562 punch from 0x01146a to 0x01ba3f (0xa5d6 bytes) 000241 1713279402.773154 fallocate 0x02c837 thru 0x03bdec (0xf5b6 bytes) 000242 1713279402.774623 punch from 0x01be8e to 0x02abd3 (0xed46 bytes) 000243 1713279402.777289 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000244 1713279402.823461 MIRROR_EXTEND to 4 mirrors 000246 1713279402.945713 mapwrite 0x00e1e9 thru 0x00f9cf (0x17e7 bytes) 000247 1713279402.975889 mapwrite 0x03152e thru 0x03ffff (0xead2 bytes) 000248 1713279402.985315 read 0x00b3d7 thru 0x016936 (0xb560 bytes) 000249 1713279402.986974 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000250 1713279403.029621 mapwrite 0x0251ee thru 0x032487 (0xd29a bytes) 000251 1713279403.041359 punch from 0x02e6b0 to 0x03677a (0x80cb bytes) 000252 1713279403.042809 punch from 0x017726 to 0x023a1f (0xc2fa bytes) 000253 1713279403.043999 mapwrite 0x004125 thru 0x0080d7 (0x3fb3 bytes) 000254 1713279403.050243 punch from 0x034f4d to 0x03ffff (0xb0b3 bytes) 000255 1713279403.051612 mapwrite 0x0006ad thru 0x003125 (0x2a79 bytes) 000256 1713279403.056518 mapread 0x00d18b thru 0x01572c (0x85a2 bytes) 000257 1713279403.058864 fallocate 0x032b80 thru 0x03cc49 (0xa0ca bytes) 000258 1713279403.060169 fallocate 0x02e4fa thru 0x0359c1 (0x74c8 bytes) 000260 1713279403.062893 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000261 1713279403.101691 punch from 0x03bdfa to 0x03ffff (0x4206 bytes) 000262 1713279403.102995 trunc from 0x040000 to 0x03b380 (0x4c7f bytes) 000263 1713279403.110639 mapwrite 0x0354af thru 0x03f1f3 (0x9d45 bytes) 000264 1713279403.124577 write 0x021848 thru 0x02d045 (0xb7fe bytes) 000266 1713279403.128625 mapread 0x0085cf thru 0x00c1fc (0x3c2e bytes) 000270 1713279403.131817 punch from 0x01b20c to 0x01bda1 (0xb96 bytes) 000272 1713279403.133600 read 0x02c3ed thru 0x038a6c (0xc680 bytes) 000273 1713279403.135455 trunc from 0x03f1f4 to 0x0391a9 (0x604a bytes) 000276 1713279403.139570 write 0x021301 thru 0x02a9c5 (0x96c5 bytes) 000277 1713279403.140388 read 0x012c81 thru 0x0186f1 (0x5a71 bytes) 000278 1713279403.144259 MIRROR_EXTEND to 3 mirrors 000279 1713279403.259896 mapwrite 0x01bff3 thru 0x0252e8 (0x92f6 bytes) 000281 1713279403.290464 fallocate 0x02369b thru 0x023b65 (0x4cb bytes) 000283 1713279403.292389 mapwrite 0x01b640 thru 0x021d2a (0x66eb bytes) 000284 1713279403.297680 mapread 0x0101f0 thru 0x01f5c1 (0xf3d2 bytes) 000285 1713279403.300300 fallocate 0x02eabd thru 0x037509 (0x8a4d bytes) 000286 1713279403.301812 trunc from 0x0391aa to 0x0303b2 (0x8df7 bytes) 000287 1713279403.311646 fallocate 0x020010 thru 0x024bb9 (0x4baa bytes) 000288 1713279403.313088 mapwrite 0x03ee11 thru 0x03ffff (0x11ef bytes) 000290 1713279403.322290 write 0x015904 thru 0x01c3e5 (0x6ae2 bytes) 000291 1713279403.323278 trunc from 0x040000 to 0x008e7b (0x37184 bytes) 000292 1713279403.327513 trunc from 0x008e7c to 0x00620f (0x2c6c bytes) 000293 1713279403.332930 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000295 1713279403.380573 mapwrite 0x024aa4 thru 0x02e432 (0x998f bytes) 000296 1713279403.396071 punch from 0x01db36 to 0x029ae7 (0xbfb2 bytes) 000297 1713279403.397950 mapwrite 0x00635c thru 0x0141dd (0xde82 bytes) 000298 1713279403.407943 punch from 0x0087de to 0x016d74 (0xe597 bytes) 000299 1713279403.409845 fallocate 0x038bac thru 0x03ffff (0x7454 bytes) 000300 1713279403.411488 mapread 0x012b07 thru 0x0215d9 (0xead3 bytes) 000304 1713279403.416830 punch from 0x02126d to 0x0268ce (0x5662 bytes) 000305 1713279403.418628 write 0x000117 thru 0x00f2ef (0xf1d9 bytes) 000307 1713279403.421546 read 0x03350a thru 0x03f64a (0xc141 bytes) 000308 1713279403.423905 fallocate 0x0262b4 thru 0x02bf54 (0x5ca1 bytes) 000309 1713279403.425417 mapwrite 0x03ed76 thru 0x03ffff (0x128a bytes) 000310 1713279403.429851 read 0x0087bb thru 0x014367 (0xbbad bytes) 000311 1713279403.430384 mapread 0x02465b thru 0x02fb84 (0xb52a bytes) 000312 1713279403.432756 read 0x0215f3 thru 0x02fdaa (0xe7b8 bytes) 000313 1713279403.433147 mapread 0x00d5bf thru 0x00e0c5 (0xb07 bytes) 000315 1713279403.433986 mapread 0x01d3c3 thru 0x02aa5f (0xd69d bytes) 000316 1713279403.434706 mapread 0x03753c thru 0x03bab5 (0x457a bytes) 000318 1713279403.435434 fallocate 0x000f86 thru 0x00b852 (0xa8cd bytes) 000319 1713279403.437048 mapread 0x0073de thru 0x007415 (0x38 bytes) 000321 1713279403.437701 mapread 0x01b216 thru 0x01d5ec (0x23d7 bytes) 000322 1713279403.438064 mapread 0x03e02e thru 0x03ffff (0x1fd2 bytes) 000323 1713279403.438487 trunc from 0x040000 to 0x013a1b (0x2c5e4 bytes) 000324 1713279403.443450 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000327 1713279403.469644 mapwrite 0x02924b thru 0x02d49b (0x4251 bytes) 000328 1713279403.483484 write 0x0341a0 thru 0x0355d6 (0x1437 bytes) 000329 1713279403.484091 trunc from 0x0355d7 to 0x02d49b (0x813b bytes) 000330 1713279403.487694 mapread 0x007633 thru 0x011ae2 (0xa4b0 bytes) 000331 1713279403.492168 write 0x01c8e2 thru 0x01cd75 (0x494 bytes) 000332 1713279403.494644 write 0x01bfe9 thru 0x01d97e (0x1996 bytes) 000337 1713279403.496382 mapread 0x004c89 thru 0x0139a1 (0xed19 bytes) 000338 1713279403.496900 trunc from 0x02d49c to 0x01c110 (0x1138b bytes) 000339 1713279403.501006 write 0x0365fd thru 0x03d2f7 (0x6cfb bytes) 000340 1713279403.501903 fallocate 0x02ddd0 thru 0x037ff4 (0xa225 bytes) 000341 1713279403.503460 read 0x028374 thru 0x03152a (0x91b7 bytes) 000342 1713279403.505878 mapread 0x00521b thru 0x008c6f (0x3a55 bytes) 000343 1713279403.506423 punch from 0x013e7a to 0x022448 (0xe5cf bytes) 000344 1713279403.510654 punch from 0x0225b1 to 0x029f1f (0x796f bytes) 000345 1713279403.512218 punch from 0x03055f to 0x03d2f7 (0xcd99 bytes) 000346 1713279403.513663 fallocate 0x014673 thru 0x0158fb (0x1289 bytes) 000347 1713279403.514955 punch from 0x012625 to 0x0222c8 (0xfca4 bytes) 000348 1713279403.516268 mapread 0x00b725 thru 0x013023 (0x78ff bytes) 000350 1713279403.519128 write 0x038666 thru 0x03bd03 (0x369e bytes) 000353 1713279403.523585 MIRROR_EXTEND to 2 mirrors 000354 1713279403.634806 MIRROR_RESYNC 2 mirrors 000355 1713279403.667737 mapread 0x00b35b thru 0x016f01 (0xbba7 bytes) 000356 1713279403.673921 write 0x0143db thru 0x0186ae (0x42d4 bytes) 000357 1713279403.683219 mapwrite 0x022c6f thru 0x027b26 (0x4eb8 bytes) 000358 1713279403.690390 mapread 0x02e907 thru 0x036b70 (0x826a bytes) 000359 1713279403.693890 punch from 0x02ddfd to 0x03bdeb (0xdfef bytes) 000360 1713279403.695397 fallocate 0x01ecce thru 0x020cb4 (0x1fe7 bytes) 000361 1713279403.696783 mapwrite 0x0267db thru 0x02aa8a (0x42b0 bytes) 000363 1713279403.701692 fallocate 0x0159c1 thru 0x01dd28 (0x8368 bytes) 000364 1713279403.703261 trunc from 0x03d2f8 to 0x02aec0 (0x12437 bytes) 000365 1713279403.711539 mapread 0x00a337 thru 0x013cb3 (0x997d bytes) 000366 1713279403.713871 mapread 0x01a417 thru 0x01b0a1 (0xc8b bytes) 000367 1713279403.714261 mapread 0x0157fe thru 0x01d9d5 (0x81d8 bytes) 000368 1713279403.714683 mapwrite 0x03a3b6 thru 0x03ffff (0x5c4a bytes) 000370 1713279403.725087 mapwrite 0x00e1c5 thru 0x019d40 (0xbb7c bytes) 000371 1713279403.730914 read 0x00faca thru 0x01890c (0x8e43 bytes) 000372 1713279403.731269 write 0x02a55e thru 0x0386a6 (0xe149 bytes) 000374 1713279403.732757 read 0x001703 thru 0x00f65b (0xdf59 bytes) 000375 1713279403.735188 fallocate 0x034a6c thru 0x036e16 (0x23ab bytes) 000376 1713279403.736866 mapread 0x02f7cd thru 0x03208a (0x28be bytes) 000379 1713279403.737763 read 0x00b457 thru 0x018dc8 (0xd972 bytes) 000380 1713279403.738041 mapread 0x025aef thru 0x029acc (0x3fde bytes) 000381 1713279403.738558 fallocate 0x006224 thru 0x014c2c (0xea09 bytes) 000382 1713279403.739995 mapwrite 0x0007a6 thru 0x0048d8 (0x4133 bytes) 000383 1713279403.744593 punch from 0x0159bf to 0x01d515 (0x7b57 bytes) 000384 1713279403.746022 trunc from 0x040000 to 0x01fc24 (0x203db bytes) 000385 1713279403.749808 trunc from 0x01fc25 to 0x0073ca (0x1885a bytes) 000386 1713279403.753161 punch from 0x001434 to 0x0073ca (0x5f97 bytes) 000387 1713279403.754576 punch from 0x006e6f to 0x0073ca (0x55c bytes) 000388 1713279403.757263 MIRROR_EXTEND to 3 mirrors 000390 1713279403.849255 fallocate 0x00ed2c thru 0x0165a4 (0x7879 bytes) 000391 1713279403.857693 mapread 0x002648 thru 0x0073ca (0x4d83 bytes) 000392 1713279403.863228 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000393 1713279403.906492 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000394 1713279403.934685 MIRROR_EXTEND to 2 mirrors 000395 1713279404.026083 trunc from 0x0073cb to 0x00020a (0x71c0 bytes) 000396 1713279404.039456 punch from 0x000053 to 0x00020a (0x1b8 bytes) 000397 1713279404.041086 fallocate 0x005388 thru 0x009090 (0x3d09 bytes) 000399 1713279404.044793 MIRROR_EXTEND to 3 mirrors 000400 1713279404.125172 mapread 0x0000c4 thru 0x00020a (0x147 bytes) 000401 1713279404.125611 read 0x000171 thru 0x00020a (0x9a bytes) 000402 1713279404.125843 trunc from 0x00020b to 0x013970 (0x13766 bytes) 000403 1713279404.135247 mapread 0x006a22 thru 0x00ef5c (0x853b bytes) 000404 1713279404.139561 trunc from 0x013971 to 0x017693 (0x3d23 bytes) 000405 1713279404.145132 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000406 1713279404.186781 mapwrite 0x03e3db thru 0x03ffff (0x1c25 bytes) 000407 1713279404.196836 punch from 0x0181b4 to 0x019334 (0x1181 bytes) 000409 1713279404.198482 fallocate 0x02de17 thru 0x032c2c (0x4e16 bytes) 000411 1713279404.200265 read 0x016ede thru 0x018c73 (0x1d96 bytes) 000412 1713279404.202965 MIRROR_EXTEND to 3 mirrors 000413 1713279404.297138 read 0x032e11 thru 0x03b330 (0x8520 bytes) 000414 1713279404.299194 mapread 0x0357ac thru 0x03ffff (0xa854 bytes) 000415 1713279404.302782 write 0x039530 thru 0x03ffff (0x6ad0 bytes) 000416 1713279404.313167 MIRROR_RESYNC 3 mirrors 000417 1713279404.408788 write 0x03da4e thru 0x03ffff (0x25b2 bytes) 000418 1713279404.418253 mapwrite 0x0308d1 thru 0x03c60d (0xbd3d bytes) 000420 1713279404.428940 mapread 0x0233ef thru 0x02ce51 (0x9a63 bytes) 000421 1713279404.432556 MIRROR_RESYNC 3 mirrors 000423 1713279404.519377 punch from 0x00a00b to 0x019b08 (0xfafe bytes) 000424 1713279404.526731 trunc from 0x040000 to 0x007735 (0x388ca bytes) 000425 1713279404.536462 MIRROR_EXTEND to 4 mirrors 000427 1713279404.623988 write 0x00978c thru 0x0103e8 (0x6c5d bytes) 000429 1713279404.632637 fallocate 0x035601 thru 0x03a835 (0x5235 bytes) 000430 1713279404.634194 mapwrite 0x02c524 thru 0x038e8f (0xc96c bytes) 000431 1713279404.654254 punch from 0x033bf0 to 0x034dbd (0x11ce bytes) 000432 1713279404.655593 mapread 0x015aad thru 0x0213bc (0xb910 bytes) 000435 1713279404.660365 fallocate 0x017a2c thru 0x019d11 (0x22e6 bytes) 000436 1713279404.662081 mapread 0x016a21 thru 0x0177a8 (0xd88 bytes) 000437 1713279404.662589 trunc from 0x038e90 to 0x032188 (0x6d07 bytes) 000438 1713279404.666347 mapwrite 0x01ec11 thru 0x026049 (0x7439 bytes) 000439 1713279404.672302 punch from 0x02ecbd to 0x032188 (0x34cc bytes) 000440 1713279404.673667 fallocate 0x0193c1 thru 0x01c784 (0x33c4 bytes) 000441 1713279404.674889 mapwrite 0x01d24a thru 0x023c96 (0x6a4d bytes) 000444 1713279404.680085 write 0x0159fa thru 0x018e35 (0x343c bytes) 000445 1713279404.681821 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000447 1713279404.723877 trunc from 0x032189 to 0x006343 (0x2be45 bytes) 000448 1713279404.729435 MIRROR_RESYNC 3 mirrors 000449 1713279404.817623 fallocate 0x036d6e thru 0x03ffff (0x9292 bytes) 000450 1713279404.826919 MIRROR_EXTEND to 4 mirrors 000451 1713279404.919345 punch from 0x00272d to 0x006343 (0x3c17 bytes) 000453 1713279404.926795 mapwrite 0x03c7a3 thru 0x03ffff (0x385d bytes) 000454 1713279404.942255 trunc from 0x040000 to 0x002e0a (0x3d1f5 bytes) 000455 1713279404.945903 mapwrite 0x0289fd thru 0x030cb2 (0x82b6 bytes) 000457 1713279404.956966 fallocate 0x00f227 thru 0x012c81 (0x3a5b bytes) 000458 1713279404.958268 fallocate 0x0123fa thru 0x015bc6 (0x37cd bytes) 000459 1713279404.959667 mapread 0x02f003 thru 0x030cb2 (0x1cb0 bytes) 000460 1713279404.960051 fallocate 0x02163d thru 0x0246d7 (0x309b bytes) 000461 1713279404.961499 trunc from 0x030cb3 to 0x03899b (0x7ce9 bytes) 000462 1713279404.964812 mapwrite 0x01f355 thru 0x02518c (0x5e38 bytes) 000465 1713279404.972292 mapwrite 0x021d29 thru 0x02550d (0x37e5 bytes) 000466 1713279404.976291 read 0x029241 thru 0x02f9a5 (0x6765 bytes) 000467 1713279404.976668 write 0x03fe14 thru 0x03ffff (0x1ec bytes) 000468 1713279404.977152 read 0x00b9d5 thru 0x00dfd5 (0x2601 bytes) 000469 1713279404.979696 trunc from 0x040000 to 0x03cddd (0x3222 bytes) 000470 1713279404.984767 MIRROR_EXTEND to 5 mirrors 000471 1713279405.101481 punch from 0x01883c to 0x01a7ef (0x1fb4 bytes) 000473 1713279405.111463 read 0x0032c9 thru 0x00a0b7 (0x6def bytes) 000474 1713279405.114128 punch from 0x01c37b to 0x01fe1c (0x3aa2 bytes) 000475 1713279405.115688 mapwrite 0x0355e3 thru 0x039aa7 (0x44c5 bytes) 000476 1713279405.125113 fallocate 0x033970 thru 0x03813f (0x47d0 bytes) 000477 1713279405.126575 write 0x01c4aa thru 0x02a8bb (0xe412 bytes) 000478 1713279405.131403 mapread 0x03487c thru 0x038e5c (0x45e1 bytes) 000479 1713279405.131828 mapwrite 0x001b2e thru 0x00fbab (0xe07e bytes) 000480 1713279405.139613 write 0x0060b8 thru 0x0135fa (0xd543 bytes) 000481 1713279405.142057 trunc from 0x03cdde to 0x010d1e (0x2c0bf bytes) 000482 1713279405.147042 mapread 0x00b690 thru 0x010d1e (0x568f bytes) 000483 1713279405.147578 mapread 0x010931 thru 0x010d1e (0x3ee bytes) 000485 1713279405.148202 write 0x03744b thru 0x038336 (0xeec bytes) 000486 1713279405.148727 write 0x030d11 thru 0x03fd29 (0xf019 bytes) 000487 1713279405.151300 write 0x01cc72 thru 0x020c05 (0x3f94 bytes) 000488 1713279405.153287 write 0x011142 thru 0x01d4a6 (0xc365 bytes) 000489 1713279405.154959 mapread 0x014ce3 thru 0x0180d2 (0x33f0 bytes) 000490 1713279405.155378 punch from 0x03b1e3 to 0x03fd29 (0x4b47 bytes) 000491 1713279405.159128 mapread 0x016e54 thru 0x017b71 (0xd1e bytes) 000492 1713279405.159539 write 0x00d12e thru 0x017384 (0xa257 bytes) 000493 1713279405.160130 read 0x01538c thru 0x024e2f (0xfaa4 bytes) 000494 1713279405.160469 punch from 0x01e258 to 0x0227c8 (0x4571 bytes) 000495 1713279405.161723 read 0x0202cd thru 0x023ef0 (0x3c24 bytes) 000496 1713279405.163947 write 0x002104 thru 0x005b61 (0x3a5e bytes) 000497 1713279405.164470 write 0x0104d8 thru 0x01a4b1 (0x9fda bytes) 000498 1713279405.166270 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000499 1713279405.211186 mapread 0x03ecbc thru 0x03fd29 (0x106e bytes) 000500 1713279405.214186 fallocate 0x003f9d thru 0x01059c (0xc600 bytes) 000501 1713279405.215861 mapwrite 0x0389ff thru 0x03fce9 (0x72eb bytes) 000503 1713279405.224964 trunc from 0x03fd2a to 0x011f73 (0x2ddb6 bytes) 000504 1713279405.231790 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000505 1713279405.272066 punch from 0x00b86d to 0x011f73 (0x6707 bytes) 000507 1713279405.273837 mapwrite 0x03ad15 thru 0x03f162 (0x444e bytes) 000510 1713279405.288203 punch from 0x0155d2 to 0x023827 (0xe256 bytes) 000511 1713279405.289626 mapread 0x034c8e thru 0x03d0f7 (0x846a bytes) 000512 1713279405.292113 trunc from 0x03f163 to 0x005946 (0x3981c bytes) 000513 1713279405.296300 fallocate 0x00f2e3 thru 0x0163a4 (0x70c2 bytes) 000514 1713279405.297924 trunc from 0x005947 to 0x017299 (0x11953 bytes) 000515 1713279405.301359 trunc from 0x01729a to 0x037b8c (0x208f3 bytes) 000516 1713279405.304584 mapread 0x004bc4 thru 0x007769 (0x2ba6 bytes) 000517 1713279405.307201 fallocate 0x0259b4 thru 0x027d81 (0x23ce bytes) 000519 1713279405.309195 trunc from 0x037b8d to 0x028703 (0xf489 bytes) 000520 1713279405.312499 read 0x004da9 thru 0x012b16 (0xdd6e bytes) 000521 1713279405.314045 punch from 0x005d6a to 0x00b459 (0x56f0 bytes) 000522 1713279405.315395 trunc from 0x028704 to 0x03ad61 (0x1265e bytes) 000523 1713279405.318738 punch from 0x02bbac to 0x03550b (0x9960 bytes) 000524 1713279405.319836 punch from 0x018b36 to 0x01dd8d (0x5258 bytes) 000527 1713279405.321561 mapwrite 0x02b111 thru 0x033760 (0x8650 bytes) 000528 1713279405.329386 write 0x004936 thru 0x009fd3 (0x569e bytes) 000529 1713279405.331161 write 0x024528 thru 0x02e278 (0x9d51 bytes) 000530 1713279405.331897 write 0x004fd2 thru 0x011b67 (0xcb96 bytes) 000532 1713279405.333112 trunc from 0x03ad62 to 0x000792 (0x3a5cf bytes) 000533 1713279405.338001 MIRROR_EXTEND to 4 mirrors 000535 1713279405.418416 mapread 0x000292 thru 0x000792 (0x501 bytes) 000536 1713279405.421487 MIRROR_RESYNC 4 mirrors 000537 1713279405.493472 fallocate 0x01e466 thru 0x02197f (0x351a bytes) 000538 1713279405.501520 read 0x0001a0 thru 0x000792 (0x5f3 bytes) 000540 1713279405.503481 punch from 0x00057b to 0x000792 (0x218 bytes) 000541 1713279405.504726 trunc from 0x000793 to 0x00dd70 (0xd5de bytes) 000542 1713279405.512344 trunc from 0x00dd71 to 0x030446 (0x226d6 bytes) 000543 1713279405.515810 write 0x02de39 thru 0x030a8a (0x2c52 bytes) 000544 1713279405.518214 trunc from 0x030a8b to 0x037fcf (0x7545 bytes) 000545 1713279405.521437 write 0x0247b6 thru 0x02fe53 (0xb69e bytes) 000546 1713279405.524137 mapread 0x031a81 thru 0x035210 (0x3790 bytes) 000547 1713279405.526311 fallocate 0x031a0b thru 0x034427 (0x2a1d bytes) 000548 1713279405.529290 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000549 1713279405.578720 fallocate 0x027ec6 thru 0x0341c3 (0xc2fe bytes) 000550 1713279405.580342 mapwrite 0x0272cc thru 0x02a946 (0x367b bytes) 000552 1713279405.589666 read 0x014528 thru 0x018509 (0x3fe2 bytes) 000553 1713279405.591779 trunc from 0x037fd0 to 0x004c05 (0x333ca bytes) 000555 1713279405.596654 mapwrite 0x0231d5 thru 0x02796e (0x479a bytes) 000557 1713279405.608118 fallocate 0x039be5 thru 0x03ffff (0x641b bytes) 000558 1713279405.609594 punch from 0x00fab9 to 0x0109cf (0xf17 bytes) 000559 1713279405.610706 read 0x035477 thru 0x03aca3 (0x582d bytes) 000560 1713279405.613928 punch from 0x02c6fc to 0x02d09e (0x9a3 bytes) 000561 1713279405.615387 fallocate 0x029a7e thru 0x02aeec (0x146f bytes) 000562 1713279405.616894 trunc from 0x040000 to 0x007f27 (0x380d8 bytes) 000564 1713279405.622427 MIRROR_RESYNC 3 mirrors 000565 1713279405.709540 punch from 0x00028c to 0x007f27 (0x7c9c bytes) 000566 1713279405.717896 mapwrite 0x0212bc thru 0x021785 (0x4ca bytes) 000567 1713279405.730558 mapread 0x012719 thru 0x01d387 (0xac6f bytes) 000568 1713279405.735446 mapwrite 0x011573 thru 0x01a95e (0x93ec bytes) 000569 1713279405.740254 punch from 0x00d13d to 0x01990d (0xc7d1 bytes) 000570 1713279405.741743 mapread 0x00dc17 thru 0x00dd39 (0x123 bytes) 000571 1713279405.744051 read 0x0131a7 thru 0x018221 (0x507b bytes) 000572 1713279405.745629 write 0x00cb40 thru 0x00fae4 (0x2fa5 bytes) 000573 1713279405.746135 write 0x00af07 thru 0x0193f7 (0xe4f1 bytes) 000574 1713279405.746869 punch from 0x01ef68 to 0x021785 (0x281e bytes) 000577 1713279405.748539 fallocate 0x000c06 thru 0x0023fa (0x17f5 bytes) 000578 1713279405.749788 fallocate 0x0068ad thru 0x00bf93 (0x56e7 bytes) 000579 1713279405.752240 MIRROR_EXTEND to 4 mirrors 000580 1713279405.850680 mapwrite 0x02b4fb thru 0x035016 (0x9b1c bytes) 000582 1713279405.873684 read 0x000b98 thru 0x00ad09 (0xa172 bytes) 000583 1713279405.875917 trunc from 0x035017 to 0x03612c (0x1116 bytes) 000584 1713279405.880733 mapread 0x026088 thru 0x02a6e2 (0x465b bytes) 000585 1713279405.881386 read 0x012bcd thru 0x01e621 (0xba55 bytes) 000586 1713279405.883959 punch from 0x009deb to 0x00ea69 (0x4c7f bytes) 000587 1713279405.885524 punch from 0x007ff6 to 0x010d04 (0x8d0f bytes) 000588 1713279405.886926 fallocate 0x02e49e thru 0x03a25a (0xbdbd bytes) 000589 1713279405.888257 punch from 0x006b9a to 0x00fde5 (0x924c bytes) 000591 1713279405.890105 mapwrite 0x02cf9a thru 0x032d9c (0x5e03 bytes) 000592 1713279405.896339 write 0x028d67 thru 0x02b831 (0x2acb bytes) 000593 1713279405.896859 fallocate 0x003f9f thru 0x00b5e0 (0x7642 bytes) 000594 1713279405.898270 fallocate 0x0394e1 thru 0x03ffff (0x6b1f bytes) 000597 1713279405.901475 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000598 1713279405.946458 punch from 0x007ab9 to 0x011327 (0x986f bytes) 000599 1713279405.948182 fallocate 0x008878 thru 0x011ec6 (0x964f bytes) 000600 1713279405.949568 trunc from 0x03a25b to 0x03f552 (0x52f8 bytes) 000601 1713279405.953707 trunc from 0x03f553 to 0x03e5e4 (0xf6e bytes) 000603 1713279405.958489 MIRROR_EXTEND to 4 mirrors 000604 1713279406.078798 write 0x0067c5 thru 0x01674c (0xff88 bytes) 000606 1713279406.090294 fallocate 0x030bd7 thru 0x03ef5f (0xe389 bytes) 000607 1713279406.091917 fallocate 0x00da79 thru 0x0153ac (0x7934 bytes) 000608 1713279406.093239 punch from 0x00dc5f to 0x00dcbe (0x60 bytes) 000609 1713279406.096509 write 0x027e4d thru 0x02ca13 (0x4bc7 bytes) 000610 1713279406.099261 read 0x033b29 thru 0x03e5e4 (0xaabc bytes) 000611 1713279406.101549 write 0x01aba8 thru 0x024726 (0x9b7f bytes) 000612 1713279406.104367 mapread 0x009003 thru 0x00f747 (0x6745 bytes) 000614 1713279406.106961 mapwrite 0x00a174 thru 0x0194b3 (0xf340 bytes) 000616 1713279406.116553 punch from 0x02e1bc to 0x038a7a (0xa8bf bytes) 000617 1713279406.119566 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000618 1713279406.163844 punch from 0x032850 to 0x0361de (0x398f bytes) 000619 1713279406.165282 mapwrite 0x02bc8d thru 0x03449a (0x880e bytes) 000621 1713279406.174961 fallocate 0x01925e thru 0x02631b (0xd0be bytes) 000622 1713279406.176444 fallocate 0x01182b thru 0x020479 (0xec4f bytes) 000623 1713279406.177962 punch from 0x031f50 to 0x03dc47 (0xbcf8 bytes) 000624 1713279406.179423 mapread 0x03c1b7 thru 0x03e5e4 (0x242e bytes) 000625 1713279406.181665 trunc from 0x03e5e5 to 0x03d35a (0x128a bytes) 000626 1713279406.186052 read 0x02a5fd thru 0x02f15b (0x4b5f bytes) 000627 1713279406.186385 punch from 0x013328 to 0x01ee46 (0xbb1f bytes) 000629 1713279406.187987 read 0x022599 thru 0x0269d1 (0x4439 bytes) 000630 1713279406.190614 read 0x029b60 thru 0x034d42 (0xb1e3 bytes) 000632 1713279406.191124 fallocate 0x0120cb thru 0x01500c (0x2f42 bytes) 000633 1713279406.192490 trunc from 0x03d35b to 0x00bde8 (0x31572 bytes) 000634 1713279406.197188 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000636 1713279406.236496 write 0x03e3f2 thru 0x03f554 (0x1163 bytes) 000637 1713279406.237997 fallocate 0x02d973 thru 0x02e2d7 (0x965 bytes) 000638 1713279406.239555 read 0x01854c thru 0x021959 (0x940e bytes) 000639 1713279406.241953 fallocate 0x00b0db thru 0x00f371 (0x4297 bytes) 000640 1713279406.243411 punch from 0x00c5b6 to 0x013e8b (0x78d6 bytes) 000641 1713279406.244695 fallocate 0x000cb1 thru 0x004284 (0x35d4 bytes) 000642 1713279406.246143 mapread 0x011a0f thru 0x012903 (0xef5 bytes) 000643 1713279406.249541 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000644 1713279406.275296 fallocate 0x001371 thru 0x008e64 (0x7af4 bytes) 000646 1713279406.277466 write 0x00675a thru 0x0155dc (0xee83 bytes) 000647 1713279406.281349 read 0x02db77 thru 0x03265e (0x4ae8 bytes) 000649 1713279406.283453 mapread 0x02feb2 thru 0x03c5b2 (0xc701 bytes) 000650 1713279406.286521 trunc from 0x03f555 to 0x03b93d (0x3c17 bytes) 000651 1713279406.289778 punch from 0x03893b to 0x0392ee (0x9b4 bytes) 000652 1713279406.291060 write 0x01db3c thru 0x029d2a (0xc1ef bytes) 000653 1713279406.295891 MIRROR_EXTEND to 2 mirrors 000655 1713279406.398660 punch from 0x031fbb to 0x03a373 (0x83b9 bytes) 000656 1713279406.406166 fallocate 0x002771 thru 0x00d85c (0xb0ec bytes) 000657 1713279406.407612 trunc from 0x03b93e to 0x03ac25 (0xd18 bytes) 000659 1713279406.416191 trunc from 0x03ac26 to 0x01944d (0x217d8 bytes) 000661 1713279406.419680 read 0x00d9dd thru 0x01944d (0xba71 bytes) 000662 1713279406.421977 mapread 0x001899 thru 0x001a98 (0x200 bytes) 000663 1713279406.423606 read 0x012743 thru 0x013135 (0x9f3 bytes) 000664 1713279406.423960 write 0x039322 thru 0x03ffff (0x6cde bytes) 000665 1713279406.424793 mapwrite 0x034156 thru 0x039f72 (0x5e1d bytes) 000668 1713279406.433255 write 0x0161e1 thru 0x01734e (0x116e bytes) 000669 1713279406.433675 read 0x00c904 thru 0x01aaff (0xe1fc bytes) 000670 1713279406.433964 fallocate 0x0177d7 thru 0x023e3c (0xc666 bytes) 000671 1713279406.435426 trunc from 0x040000 to 0x01db3f (0x224c0 bytes) 000672 1713279406.438539 punch from 0x012f50 to 0x01bc41 (0x8cf2 bytes) 000673 1713279406.441288 write 0x02f2d2 thru 0x039cce (0xa9fd bytes) 000674 1713279406.442001 punch from 0x01c165 to 0x021611 (0x54ad bytes) 000675 1713279406.443615 write 0x020620 thru 0x02069f (0x80 bytes) 000676 1713279406.446416 read 0x01a6e2 thru 0x025430 (0xad4f bytes) 000677 1713279406.446878 punch from 0x02241c to 0x02be35 (0x9a1a bytes) 000678 1713279406.448190 write 0x006949 thru 0x0079b4 (0x106c bytes) 000680 1713279406.449163 mapread 0x02e16c thru 0x0301fd (0x2092 bytes) 000681 1713279406.449653 mapwrite 0x0339f1 thru 0x03787c (0x3e8c bytes) 000682 1713279406.455249 mapread 0x009964 thru 0x00ecb8 (0x5355 bytes) 000683 1713279406.455759 trunc from 0x039ccf to 0x02a16b (0xfb63 bytes) 000684 1713279406.459381 mapread 0x001dcc thru 0x00d481 (0xb6b6 bytes) 000686 1713279406.460171 write 0x03efc7 thru 0x03ffff (0x1039 bytes) 000687 1713279406.460613 mapread 0x018fff thru 0x023be5 (0xabe7 bytes) 000688 1713279406.462986 write 0x01304a thru 0x014ebc (0x1e73 bytes) 000690 1713279406.465002 MIRROR_RESYNC 2 mirrors 000691 1713279406.562415 trunc from 0x040000 to 0x03b428 (0x4bd7 bytes) 000692 1713279406.576394 MIRROR_RESYNC 2 mirrors 000693 1713279406.670031 mapread 0x01e96c thru 0x01ee73 (0x508 bytes) 000694 1713279406.671610 write 0x01525f thru 0x0195dd (0x437f bytes) 000695 1713279406.681068 read 0x00313f thru 0x008c2a (0x5aec bytes) 000696 1713279406.682969 punch from 0x00448e to 0x009821 (0x5394 bytes) 000697 1713279406.684605 mapwrite 0x024096 thru 0x0275c9 (0x3534 bytes) 000698 1713279406.692660 trunc from 0x03b429 to 0x01f46f (0x1bfb9 bytes) 000699 1713279406.700562 read 0x01c5b3 thru 0x01f46f (0x2ebd bytes) 000700 1713279406.703164 mapread 0x01c386 thru 0x01efd1 (0x2c4c bytes) 000701 1713279406.703749 mapread 0x00b588 thru 0x01180f (0x6288 bytes) 000702 1713279406.704305 mapwrite 0x013656 thru 0x017645 (0x3ff0 bytes) 000704 1713279406.709768 write 0x03e8b1 thru 0x03ffff (0x174f bytes) 000705 1713279406.710412 fallocate 0x0237e8 thru 0x02896a (0x5183 bytes) 000706 1713279406.711941 mapwrite 0x015fcc thru 0x021f94 (0xbfc9 bytes) 000708 1713279406.720765 mapwrite 0x039d64 thru 0x03ebd4 (0x4e71 bytes) 000709 1713279406.725405 punch from 0x019e3a to 0x022c05 (0x8dcc bytes) 000710 1713279406.726959 mapread 0x00271b thru 0x003c45 (0x152b bytes) 000711 1713279406.727499 mapwrite 0x01fb62 thru 0x02430f (0x47ae bytes) 000712 1713279406.735802 trunc from 0x040000 to 0x021313 (0x1ecec bytes) 000713 1713279406.739784 write 0x03cae5 thru 0x03ffff (0x351b bytes) 000714 1713279406.741348 MIRROR_RESYNC 2 mirrors 000715 1713279406.834882 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000717 1713279406.860555 write 0x020b92 thru 0x02ff0b (0xf37a bytes) 000718 1713279406.866001 mapread 0x03e150 thru 0x03ffff (0x1eb0 bytes) 000719 1713279406.868180 punch from 0x034675 to 0x039380 (0x4d0c bytes) 000720 1713279406.869569 mapread 0x03acf2 thru 0x03b02b (0x33a bytes) 000721 1713279406.870119 read 0x019bb4 thru 0x024283 (0xa6d0 bytes) 000722 1713279406.872221 mapread 0x0252fc thru 0x030ecb (0xbbd0 bytes) 000724 1713279406.874991 punch from 0x016814 to 0x01f775 (0x8f62 bytes) 000725 1713279406.876426 fallocate 0x01ac5d thru 0x0277ff (0xcba3 bytes) 000726 1713279406.877842 mapwrite 0x03d094 thru 0x03ffff (0x2f6c bytes) 000727 1713279406.882847 mapwrite 0x01fe3d thru 0x025d2c (0x5ef0 bytes) 000729 1713279406.888919 mapread 0x028cf8 thru 0x033280 (0xa589 bytes) 000730 1713279406.889355 write 0x00cf9f thru 0x00e22e (0x1290 bytes) 000731 1713279406.891777 read 0x011713 thru 0x0125bc (0xeaa bytes) 000732 1713279406.892215 fallocate 0x024322 thru 0x02ea24 (0xa703 bytes) 000733 1713279406.893988 mapwrite 0x00b8f9 thru 0x00cf0c (0x1614 bytes) 000735 1713279406.898867 MIRROR_EXTEND to 2 mirrors 000736 1713279407.003733 write 0x0166f6 thru 0x01d16a (0x6a75 bytes) 000738 1713279407.014266 punch from 0x012304 to 0x01573c (0x3439 bytes) 000739 1713279407.015549 mapread 0x0367a7 thru 0x03ffff (0x9859 bytes) 000741 1713279407.019486 write 0x005317 thru 0x00ae78 (0x5b62 bytes) 000742 1713279407.022950 MIRROR_RESYNC 2 mirrors 000743 1713279407.108765 mapread 0x027db4 thru 0x02fe75 (0x80c2 bytes) 000745 1713279407.113003 trunc from 0x040000 to 0x03d0cd (0x2f32 bytes) 000746 1713279407.124958 punch from 0x02f1ed to 0x03c537 (0xd34b bytes) 000747 1713279407.126230 trunc from 0x03d0ce to 0x031655 (0xba78 bytes) 000748 1713279407.130525 fallocate 0x00092b thru 0x008c4a (0x8320 bytes) 000749 1713279407.131928 trunc from 0x031656 to 0x01063c (0x21019 bytes) 000750 1713279407.135302 trunc from 0x01063d to 0x020a31 (0x103f5 bytes) 000752 1713279407.139229 fallocate 0x026c0f thru 0x02d021 (0x6413 bytes) 000753 1713279407.140755 write 0x00f5c8 thru 0x01f01f (0xfa58 bytes) 000754 1713279407.144541 fallocate 0x013e6b thru 0x0203a6 (0xc53c bytes) 000757 1713279407.146375 mapwrite 0x012f2a thru 0x019ee6 (0x6fbd bytes) 000758 1713279407.152068 punch from 0x00ae35 to 0x01a687 (0xf853 bytes) 000759 1713279407.153647 mapwrite 0x03a326 thru 0x03ffff (0x5cda bytes) 000760 1713279407.163823 write 0x013d48 thru 0x0155af (0x1868 bytes) 000761 1713279407.166258 mapread 0x03570a thru 0x03a7b9 (0x50b0 bytes) 000762 1713279407.166684 fallocate 0x010810 thru 0x01ac85 (0xa476 bytes) 000763 1713279407.169103 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000764 1713279407.197447 write 0x00a74c thru 0x010fd1 (0x6886 bytes) 000765 1713279407.200819 mapread 0x00b362 thru 0x014d1c (0x99bb bytes) 000768 1713279407.203337 read 0x001556 thru 0x0019d4 (0x47f bytes) 000770 1713279407.204775 write 0x012b2c thru 0x017df2 (0x52c7 bytes) 000771 1713279407.205251 punch from 0x02af01 to 0x0398a0 (0xe9a0 bytes) 000773 1713279407.206784 read 0x02d73b thru 0x0370ff (0x99c5 bytes) 000774 1713279407.210844 MIRROR_EXTEND to 2 mirrors 000775 1713279407.321062 mapread 0x02ddf7 thru 0x039b13 (0xbd1d bytes) 000776 1713279407.327650 punch from 0x01feaa to 0x02fa92 (0xfbe9 bytes) 000777 1713279407.336347 fallocate 0x00dc1a thru 0x010b66 (0x2f4d bytes) 000778 1713279407.337906 mapread 0x035b4b thru 0x03f515 (0x99cb bytes) 000780 1713279407.342534 fallocate 0x035819 thru 0x03f30e (0x9af6 bytes) 000781 1713279407.343999 write 0x037e0d thru 0x03b60c (0x3800 bytes) 000782 1713279407.345807 punch from 0x02b93a to 0x03662c (0xacf3 bytes) 000783 1713279407.348622 MIRROR_RESYNC 2 mirrors 000785 1713279407.430465 mapwrite 0x011004 thru 0x013458 (0x2455 bytes) 000786 1713279407.457023 write 0x015f74 thru 0x01ebdc (0x8c69 bytes) 000788 1713279407.460172 punch from 0x024270 to 0x029d8b (0x5b1c bytes) 000789 1713279407.462512 MIRROR_RESYNC 2 mirrors 000790 1713279407.543628 MIRROR_EXTEND to 3 mirrors 000791 1713279407.643621 write 0x02d1b7 thru 0x02d41c (0x266 bytes) 000792 1713279407.653017 read 0x03e2f7 thru 0x03ffff (0x1d09 bytes) 000793 1713279407.654652 write 0x038f5a thru 0x03ffff (0x70a6 bytes) 000794 1713279407.656470 fallocate 0x02018f thru 0x0282e8 (0x815a bytes) 000796 1713279407.658282 read 0x022c18 thru 0x02b806 (0x8bef bytes) 000797 1713279407.661164 fallocate 0x03c90b thru 0x03ffff (0x36f5 bytes) 000798 1713279407.662537 read 0x02a876 thru 0x02b9f2 (0x117d bytes) 000799 1713279407.662832 read 0x02d9db thru 0x030f78 (0x359e bytes) 000800 1713279407.663160 read 0x0152b3 thru 0x0206b7 (0xb405 bytes) 000801 1713279407.664697 MIRROR_EXTEND to 4 mirrors 000802 1713279407.769941 read 0x03be1c thru 0x03ffff (0x41e4 bytes) 000803 1713279407.773198 write 0x007c37 thru 0x00bc37 (0x4001 bytes) 000804 1713279407.783510 punch from 0x001b30 to 0x00f9d7 (0xdea8 bytes) 000805 1713279407.787020 write 0x02154e thru 0x026850 (0x5303 bytes) 000806 1713279407.789410 write 0x01d19a thru 0x0282eb (0xb152 bytes) 000808 1713279407.790623 punch from 0x03cab5 to 0x03ffff (0x354b bytes) 000809 1713279407.791854 trunc from 0x040000 to 0x02fcdc (0x10323 bytes) 000810 1713279407.799416 punch from 0x002915 to 0x00db36 (0xb222 bytes) 000811 1713279407.800542 read 0x008fab thru 0x009348 (0x39e bytes) 000812 1713279407.803042 trunc from 0x02fcdd to 0x004398 (0x2b944 bytes) 000813 1713279407.806768 mapread 0x00319b thru 0x004398 (0x11fe bytes) 000814 1713279407.807382 write 0x0038be thru 0x00f213 (0xb956 bytes) 000815 1713279407.808028 punch from 0x0009ff to 0x0058ef (0x4ef1 bytes) 000816 1713279407.811195 mapwrite 0x015da7 thru 0x0180ee (0x2348 bytes) 000817 1713279407.823410 read 0x005e1b thru 0x00cebb (0x70a1 bytes) 000818 1713279407.825764 read 0x00d709 thru 0x01808e (0xa986 bytes) 000820 1713279407.826365 mapwrite 0x02c2d3 thru 0x03489b (0x85c9 bytes) 000821 1713279407.838370 fallocate 0x018798 thru 0x025729 (0xcf92 bytes) 000822 1713279407.839810 write 0x019c93 thru 0x01eba1 (0x4f0f bytes) 000824 1713279407.840729 mapread 0x01fb22 thru 0x02fa6c (0xff4b bytes) 000825 1713279407.842603 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000826 1713279407.886025 trunc from 0x03489c to 0x03ab5f (0x62c4 bytes) 000827 1713279407.890358 punch from 0x0071b9 to 0x009077 (0x1ebf bytes) 000828 1713279407.891739 fallocate 0x02a29d thru 0x036337 (0xc09b bytes) 000829 1713279407.893001 trunc from 0x03ab60 to 0x03b255 (0x6f6 bytes) 000830 1713279407.896356 punch from 0x01a853 to 0x028a8c (0xe23a bytes) 000832 1713279407.897972 mapread 0x00d530 thru 0x01d23a (0xfd0b bytes) 000833 1713279407.903719 trunc from 0x03b256 to 0x01432a (0x26f2b bytes) 000834 1713279407.906888 mapread 0x012c2d thru 0x01432a (0x16fe bytes) 000837 1713279407.907923 mapread 0x0118bc thru 0x01432a (0x2a6f bytes) 000838 1713279407.908417 write 0x02a20d thru 0x03364d (0x9441 bytes) 000840 1713279407.909487 fallocate 0x01b897 thru 0x02b1d6 (0xf940 bytes) 000841 1713279407.910991 trunc from 0x03364e to 0x03ce62 (0x9815 bytes) 000842 1713279407.914057 trunc from 0x03ce63 to 0x0155b3 (0x278af bytes) 000843 1713279407.917281 write 0x023c68 thru 0x031272 (0xd60b bytes) 000844 1713279407.918218 read 0x0293ed thru 0x031272 (0x7e86 bytes) 000845 1713279407.918511 fallocate 0x00e4de thru 0x0160b7 (0x7bda bytes) 000846 1713279407.919924 punch from 0x01ca8d to 0x025acb (0x903f bytes) 000847 1713279407.923521 punch from 0x01d28a to 0x0297f8 (0xc56f bytes) 000848 1713279407.924729 mapwrite 0x03da2f thru 0x03f8cb (0x1e9d bytes) 000849 1713279407.934021 read 0x00c4ac thru 0x01acc5 (0xe81a bytes) 000851 1713279407.936883 mapwrite 0x01ce8d thru 0x029ecc (0xd040 bytes) 000852 1713279407.942021 trunc from 0x03f8cc to 0x002e4b (0x3ca80 bytes) 000853 1713279407.945779 fallocate 0x01e295 thru 0x028596 (0xa302 bytes) 000855 1713279407.948579 MIRROR_RESYNC 3 mirrors 000856 1713279408.038599 mapwrite 0x007bd3 thru 0x00e094 (0x64c2 bytes) 000857 1713279408.056561 mapwrite 0x033eb0 thru 0x03ffff (0xc150 bytes) 000858 1713279408.071224 MIRROR_RESYNC 3 mirrors 000859 1713279408.180503 write 0x02334e thru 0x02ede7 (0xba9a bytes) 000861 1713279408.190392 fallocate 0x0380f1 thru 0x03ffff (0x7f0f bytes) 000862 1713279408.192180 punch from 0x0313a4 to 0x03393b (0x2598 bytes) 000863 1713279408.193551 punch from 0x01d670 to 0x023438 (0x5dc9 bytes) 000864 1713279408.196951 write 0x001f97 thru 0x006cf0 (0x4d5a bytes) 000865 1713279408.199597 read 0x0360d4 thru 0x03ffff (0x9f2c bytes) 000867 1713279408.201862 read 0x00ead2 thru 0x0154ea (0x6a19 bytes) 000868 1713279408.204470 read 0x018d1e thru 0x024208 (0xb4eb bytes) 000869 1713279408.204785 punch from 0x031b65 to 0x036f77 (0x5413 bytes) 000871 1713279408.206543 write 0x00d160 thru 0x016a42 (0x98e3 bytes) 000872 1713279408.208579 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000873 1713279408.253894 punch from 0x001d14 to 0x009bf8 (0x7ee5 bytes) 000874 1713279408.255511 mapwrite 0x026240 thru 0x0352ec (0xf0ad bytes) 000875 1713279408.268732 punch from 0x03177e to 0x035341 (0x3bc4 bytes) 000876 1713279408.270630 write 0x019561 thru 0x020f71 (0x7a11 bytes) 000877 1713279408.274556 write 0x021444 thru 0x025a14 (0x45d1 bytes) 000879 1713279408.275262 mapread 0x001c4f thru 0x009333 (0x76e5 bytes) 000880 1713279408.278983 punch from 0x01bb15 to 0x0208e5 (0x4dd1 bytes) 000881 1713279408.282144 mapwrite 0x03fd76 thru 0x03ffff (0x28a bytes) 000882 1713279408.286843 mapwrite 0x03b0a7 thru 0x03ffff (0x4f59 bytes) 000883 1713279408.292692 MIRROR_RESYNC 2 mirrors 000884 1713279408.404889 punch from 0x039486 to 0x03a6b5 (0x1230 bytes) 000885 1713279408.412608 trunc from 0x040000 to 0x00ab4b (0x354b4 bytes) 000886 1713279408.423227 MIRROR_RESYNC 2 mirrors 000888 1713279408.494885 fallocate 0x02e635 thru 0x03d04e (0xea1a bytes) 000889 1713279408.502308 write 0x01531a thru 0x0194cd (0x41b4 bytes) 000890 1713279408.504645 write 0x02be73 thru 0x0300dd (0x426b bytes) 000891 1713279408.506790 fallocate 0x01d4f7 thru 0x01e88f (0x1399 bytes) 000892 1713279408.508377 mapwrite 0x01fd41 thru 0x02c925 (0xcbe5 bytes) 000893 1713279408.517256 MIRROR_EXTEND to 3 mirrors 000894 1713279408.628618 mapwrite 0x0206a7 thru 0x0239cb (0x3325 bytes) 000895 1713279408.655349 read 0x0248a3 thru 0x02f5c6 (0xad24 bytes) 000897 1713279408.657794 write 0x037244 thru 0x03ffff (0x8dbc bytes) 000898 1713279408.661271 mapread 0x022a0f thru 0x022df7 (0x3e9 bytes) 000899 1713279408.661817 trunc from 0x040000 to 0x008f30 (0x370cf bytes) 000900 1713279408.670623 fallocate 0x014d3e thru 0x02184b (0xcb0e bytes) 000901 1713279408.672342 fallocate 0x01a2b7 thru 0x027aa6 (0xd7f0 bytes) 000902 1713279408.673946 trunc from 0x027aa7 to 0x0235cf (0x44d7 bytes) 000903 1713279408.678689 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000905 1713279408.721579 fallocate 0x03740f thru 0x03cbfe (0x57f0 bytes) 000907 1713279408.723708 trunc from 0x03cbff to 0x009074 (0x33b8a bytes) 000908 1713279408.728307 mapwrite 0x00fbd9 thru 0x01307d (0x34a5 bytes) 000910 1713279408.741227 read 0x007a7f thru 0x0094ab (0x1a2d bytes) 000911 1713279408.742847 trunc from 0x01307e to 0x02556e (0x124f1 bytes) 000912 1713279408.746448 read 0x01cd56 thru 0x02556e (0x8819 bytes) 000913 1713279408.748926 mapwrite 0x02b7b5 thru 0x033dfe (0x864a bytes) 000914 1713279408.759734 trunc from 0x033dff to 0x022a45 (0x113b9 bytes) 000915 1713279408.763144 mapread 0x013634 thru 0x01933a (0x5d07 bytes) 000917 1713279408.763948 punch from 0x014cc6 to 0x017521 (0x285c bytes) 000918 1713279408.765247 mapread 0x007e96 thru 0x00b21d (0x3388 bytes) 000919 1713279408.766926 MIRROR_RESYNC 2 mirrors 000920 1713279408.847990 fallocate 0x025665 thru 0x02b297 (0x5c33 bytes) 000921 1713279408.855144 trunc from 0x02b298 to 0x036171 (0xaeda bytes) 000922 1713279408.862687 write 0x03dbf8 thru 0x03ffff (0x2408 bytes) 000923 1713279408.864783 MIRROR_EXTEND to 3 mirrors 000924 1713279408.968600 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000925 1713279409.009920 write 0x002459 thru 0x009999 (0x7541 bytes) 000926 1713279409.017281 MIRROR_EXTEND to 3 mirrors 000927 1713279409.132373 mapread 0x02af6e thru 0x02c6b3 (0x1746 bytes) 000928 1713279409.137851 MIRROR_EXTEND to 4 mirrors 000929 1713279409.267430 write 0x0257cd thru 0x02d9f0 (0x8224 bytes) 000930 1713279409.278261 read 0x013d14 thru 0x015197 (0x1484 bytes) 000931 1713279409.279813 trunc from 0x040000 to 0x02af44 (0x150bb bytes) 000932 1713279409.284456 trunc from 0x02af45 to 0x027d54 (0x31f0 bytes) 000933 1713279409.288984 MIRROR_EXTEND to 5 mirrors 000934 1713279409.394382 mapread 0x007b36 thru 0x0132e5 (0xb7b0 bytes) 000935 1713279409.404042 mapwrite 0x003249 thru 0x005a3d (0x27f5 bytes) 000941 1713279409.435193 trunc from 0x027d55 to 0x006f15 (0x20e3f bytes) 000942 1713279409.444581 fallocate 0x0342e6 thru 0x03ffff (0xbd1a bytes) 000943 1713279409.446091 trunc from 0x006f16 to 0x02fb5a (0x28c45 bytes) 000944 1713279409.449791 punch from 0x01bb21 to 0x021712 (0x5bf2 bytes) 000945 1713279409.451251 fallocate 0x01adef thru 0x02a69b (0xf8ad bytes) 000946 1713279409.452855 mapwrite 0x02582a thru 0x0318cb (0xc0a2 bytes) 000947 1713279409.468867 MIRROR_RESYNC 5 mirrors 000948 1713279409.613819 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000949 1713279409.673743 MIRROR_RESYNC 4 mirrors 000950 1713279409.716831 mapwrite 0x02c0e0 thru 0x02e9f8 (0x2919 bytes) 000951 1713279409.744474 fallocate 0x018a7f thru 0x01e15c (0x56de bytes) 000952 1713279409.747110 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000953 1713279409.789307 punch from 0x013807 to 0x017337 (0x3b31 bytes) 000957 1713279409.791588 fallocate 0x017447 thru 0x01b747 (0x4301 bytes) 000958 1713279409.792991 read 0x01ec5e thru 0x0268b8 (0x7c5b bytes) 000960 1713279409.795704 punch from 0x01b2e5 to 0x028962 (0xd67e bytes) 000961 1713279409.797082 trunc from 0x0318cc to 0x039ec0 (0x85f5 bytes) 000962 1713279409.804241 trunc from 0x039ec1 to 0x018fad (0x20f13 bytes) 000963 1713279409.807576 punch from 0x0127bb to 0x018fad (0x67f3 bytes) 000964 1713279409.808807 write 0x0245dc thru 0x028e58 (0x487d bytes) 000966 1713279409.809744 write 0x0314fa thru 0x03fa3b (0xe542 bytes) 000967 1713279409.810579 mapread 0x02fd85 thru 0x038b90 (0x8e0c bytes) 000968 1713279409.813344 trunc from 0x03fa3c to 0x02d1fd (0x1283e bytes) 000969 1713279409.816945 write 0x02d115 thru 0x03a5f3 (0xd4df bytes) 000970 1713279409.820461 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000971 1713279409.862854 fallocate 0x02b727 thru 0x02ea0a (0x32e4 bytes) 000972 1713279409.864561 write 0x00bd45 thru 0x01a1f5 (0xe4b1 bytes) 000974 1713279409.870248 MIRROR_RESYNC 2 mirrors 000975 1713279409.971776 read 0x0159b5 thru 0x01cb7a (0x71c6 bytes) 000976 1713279409.974184 punch from 0x02e81a to 0x0311c4 (0x29ab bytes) 000977 1713279409.981989 mapwrite 0x0211ee thru 0x02e7cb (0xd5de bytes) 000978 1713279409.995152 write 0x021c1c thru 0x023d9d (0x2182 bytes) 000979 1713279409.995748 read 0x01e25a thru 0x02cee1 (0xec88 bytes) 000980 1713279409.996047 write 0x01e472 thru 0x01ebfb (0x78a bytes) 000981 1713279409.996438 fallocate 0x02b9a5 thru 0x0309e6 (0x5042 bytes) 000982 1713279409.997773 fallocate 0x034d56 thru 0x03ffff (0xb2aa bytes) 000983 1713279409.999172 punch from 0x03eab7 to 0x03ffff (0x1549 bytes) 000985 1713279410.000630 punch from 0x030e7a to 0x03e1b0 (0xd337 bytes) 000986 1713279410.001843 read 0x03ffe5 thru 0x03ffff (0x1b bytes) 000987 1713279410.005583 MIRROR_EXTEND to 3 mirrors 000989 1713279410.128002 mapread 0x0190ec thru 0x019672 (0x587 bytes) 000991 1713279410.129668 mapread 0x034081 thru 0x03a23e (0x61be bytes) 000992 1713279410.132752 trunc from 0x040000 to 0x0193e7 (0x26c18 bytes) 000994 1713279410.144193 read 0x012672 thru 0x0193e7 (0x6d76 bytes) 000995 1713279410.146373 fallocate 0x03f945 thru 0x03ffff (0x6bb bytes) 000996 1713279410.147856 write 0x022a24 thru 0x023a9a (0x1077 bytes) 000997 1713279410.151033 MIRROR_EXTEND to 4 mirrors 000998 1713279410.269382 mapread 0x020592 thru 0x020a18 (0x487 bytes) 000999 1713279410.275145 MIRROR_EXTEND to 5 mirrors 001000 1713279410.388836 write 0x0230bc thru 0x0241f4 (0x1139 bytes) All operations completed A-OK! PASS 70a (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 10:56:53 (1713279413) Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre2 Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1919488 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7206880 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6442752 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8953568 .. failed Extending file size to 7980064 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8631008 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9319680 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9080032 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9118240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9409024 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 12592 12593 12595 12596 12598 failed 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) 192.168.202.159@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg259-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:57:58 (1713279478) Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre2 Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6811520 .. Extending file size to 1280384 .. failed Extending file size to 6929536 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6988800 .. Extending file size to 7152288 .. Extending file size to 7384000 .. Extending file size to 8638432 .. Extending file size to 8647008 .. done Extending file size to 7682880 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8127168 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8294112 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8685184 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8359584 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8673504 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9293216 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8908480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 16248 16249 16251 16253 16255 done 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) 192.168.202.159@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg259-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:59:03 (1713279543) Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2532 1285156 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3292 1284396 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21848 3585172 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15912 3591108 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37760 7176280 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2532 1285156 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3292 1284396 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21848 3585172 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15912 3591108 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37760 7176280 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0353005 s, 59.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0464981 s, 45.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:59:09 (1713279549) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0245707 s, 85.4 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 10:59:12 (1713279552) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:59:15 (1713279555) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0238581 s, 87.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1015:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1014:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1015:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:59:19 (1713279559) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0207232 s, 101 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 204b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:59:22 (1713279562) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.031423 s, 66.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1016:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1016:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1016:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1016:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1017:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:59:46 (1713279586) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125002 s, 83.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } PASS 204d (36s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:00:23 (1713279623) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0199584 s, 105 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:00:58 (1713279658) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool oleg259-server: Pool lustre.test_204f created Adding targets to pool oleg259-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg259-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0193639 s, 108 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1017:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1018:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1017:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1018:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg259-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg259-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg259-server: Pool lustre.test_204f destroyed PASS 204f (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 11:01:45 (1713279705) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:01:49 (1713279709) oleg259-server: Pool lustre.test_206 created Adding targets to pool oleg259-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg259-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg259-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg259-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg259-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:02:02 (1713279722) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186381 s, 56.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102187 s, 103 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xbd9a6696 Mirror 3: 0xbd9a6696 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:02:06 (1713279726) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:02:08 (1713279728) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:02:11 (1713279731) Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0074096 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x650000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x650000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x650000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c60000 Read this (4 bytes): BBBB 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:02:26 (1713279746) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00233617 s, 3.5 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 11:02:29 (1713279749) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00971496 s, 108 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611259 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-16 11:02:30.000000000 -0400 Modify: 2024-04-16 11:02:30.000000000 -0400 Change: 2024-04-16 11:02:30.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:02:33 (1713279753) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00861823 s, 122 MB/s oleg259-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg259-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg259-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg259-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg259-server: O/280000401/d24/b8: File not found by ext2_lookup PASS 210b (10s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 772 sec ============ 11:02:43 (1713279763) === sanity-flr: start cleanup 11:02:44 (1713279764) === === sanity-flr: finish cleanup 11:02:44 (1713279764) ===