-----============= acceptance-small: sanity-flr ============----- Thu Apr 18 03:23:15 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 03:23:18 (1713424998) === oleg114-client.virtnet: executing check_config_client /mnt/lustre oleg114-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg114-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b58c0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b58c0000.idle_timeout=debug disable quota as required oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 03:23:26 (1713425006) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6909] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 03:23:27 (1713425007) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:23:32 (1713425012) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:23:34 (1713425014) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 03:23:36 (1713425016) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:23:42 (1713425022) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:23:44 (1713425024) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 03:23:46 (1713425026) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b58c0000.stats=clear osc.lustre-OST0001-osc-ffff8800b58c0000.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:23:49 (1713425029) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:23:53 (1713425033) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:23:56 (1713425036) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 03:23:58 (1713425038) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 03:24:02 (1713425042) PASS 2 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:24:04 (1713425044) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:24:07 (1713425047) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:24:10 (1713425050) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 03:24:14 (1713425054) PASS 7 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:24:17 (1713425057) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.510742 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0428682 s, 24.5 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:24:21 (1713425061) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.0466055 s, 157 MB/s PASS 22 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:24:24 (1713425064) Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58c0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58c0000.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58c0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58c0000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:24:45 (1713425085) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.149586 s, 63.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58c0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58c0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:24:54 (1713425094) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58c0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58c0000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58c0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58c0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:25:10 (1713425110) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.727632 s, 72.1 MB/s -rw-r--r-- 1 root root 52428800 Apr 18 03:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.159s user 0m0.001s sys 0m0.108s Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58c0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58c0000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... real 0m0.141s user 0m0.000s sys 0m0.103s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58c0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58c0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.169s user 0m0.002s sys 0m0.135s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b58c0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58c0000.ost_server_uuid in FULL state after 0 sec PASS 33b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:25:31 (1713425131) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:25:33 (1713425133) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:25:35 (1713425135) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 03:25:37 (1713425137) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 03:25:40 (1713425140) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.113781 s, 111 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:25:45 (1713425145) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00690466 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00766455 s, 137 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:25:47 (1713425147) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:25:51 (1713425151) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000532881 s, 1.9 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000424513 s, 2.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000514113 s, 2.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000370695 s, 2.8 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 03:25:55 (1713425155) create test files with size 15752411 .. 539643b3425a617a190c3b89bc45d150 - c006866b513f09c6a852a0372bdf5f7b - 57ac8fd326e44b9fe1c18dc0493a7398 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 03:26:01 (1713425161) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:26:05 (1713425165) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:26:08 (1713425168) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199326 s, 52.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00824181 s, 127 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 03:26:11 (1713425171) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0239108 s, 87.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0781538 s, 53.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0357985 s, 58.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 03:26:15 (1713425175) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 03:26:17 (1713425177) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:26:20 (1713425180) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00699273 s, 150 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 03:26:23 (1713425183) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:26:25 (1713425185) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:26:28 (1713425188) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0529186 s, 198 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:26:32 (1713425192) llite.lustre-ffff8800b00bb000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0646273 s, 162 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800b65c6800.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:26:36 (1713425196) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 03:26:44 (1713425204) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 03:26:50 (1713425210) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 03:26:52 (1713425212) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0529113 s, 59.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0405628 s, 77.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 05e052735cb3385afdd9d87b659085a5 - mirror 1 checksum: 8619f937f1d8ce799232836c9f343689 - mirror 2 checksum: 05e052735cb3385afdd9d87b659085a5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 03:26:53 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 03:26:53 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 05e052735cb3385afdd9d87b659085a5 - mirror 1 checksum: 8619f937f1d8ce799232836c9f343689 - mirror 2 checksum: 8619f937f1d8ce799232836c9f343689 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 03:26:53 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 03:26:53 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:26:56 (1713425216) Failing mds1 on oleg114-server Stopping /mnt/lustre-mds1 (opts:) on oleg114-server 03:26:58 (1713425218) shut down Failover mds1 to oleg114-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 03:27:10 (1713425230) targets are mounted 03:27:10 (1713425230) facet_failover done oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00931165 s, 113 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:27:16 (1713425236) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 03:27:18 (1713425238) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:27:20 (1713425240) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00176406 s, 9.3 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 03:27:23 (1713425243) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:27:26 (1713425246) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:27:28 (1713425248) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:27:33 (1713425253) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:27:53 (1713425273) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:28:11 (1713425291) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:28:30 (1713425310) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 03:28:33 (1713425313) mirror create pid 26571 mirror split pid 26572 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26571 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26572 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 03:29:37 (1713425377) Seed set to 5377 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1d330 000001 1713425377.918824 trunc from 00000000 to 0x01d32f (0x1d330 bytes) 000003 1713425377.924466 mapwrite 0x038fb6 thru 0x03ffff (0x704a bytes) 000006 1713425377.977647 write 0x01994e thru 0x01cefb (0x35ae bytes) 000009 1713425377.981769 mapwrite 0x00770a thru 0x00fd0c (0x8603 bytes) 000010 1713425378.004627 read 0x026294 thru 0x02bbd8 (0x5945 bytes) 000012 1713425378.007776 mapwrite 0x03fdf2 thru 0x03ffff (0x20e bytes) 000015 1713425378.022009 write 0x03b6ec thru 0x03edd8 (0x36ed bytes) 000024 1713425378.024250 mapwrite 0x016b33 thru 0x018354 (0x1822 bytes) 000025 1713425378.041643 mapwrite 0x019d0c thru 0x01d127 (0x341c bytes) truncating to largest ever: 0x208dd 000026 1713425378.063706 trunc from 0x040000 to 0x0208dc (0x1f723 bytes) 000029 1713425378.070118 trunc from 0x0208dd to 0x0191e5 (0x76f7 bytes) 000030 1713425378.077481 MIRROR_EXTEND to 2 mirrors 000033 1713425378.198554 write 0x0135d1 thru 0x01ec07 (0xb637 bytes) truncating to largest ever: 0x21346 000034 1713425378.218003 trunc from 0x01ec08 to 0x021345 (0x273e bytes) 000035 1713425378.231255 MIRROR_RESYNC 2 mirrors 000036 1713425378.360291 read 0x01529e thru 0x021345 (0xc0a8 bytes) 000037 1713425378.364598 mapread 0x01eba4 thru 0x021345 (0x27a2 bytes) truncating to largest ever: 0x2489a 000039 1713425378.365713 trunc from 0x021346 to 0x024899 (0x3554 bytes) 000040 1713425378.400920 mapwrite 0x031b0d thru 0x039dc0 (0x82b4 bytes) 000041 1713425378.448988 write 0x033642 thru 0x03982d (0x61ec bytes) truncating to largest ever: 0x34128 000042 1713425378.450228 trunc from 0x039dc1 to 0x034127 (0x5c99 bytes) 000043 1713425378.459254 MIRROR_EXTEND to 3 mirrors 000044 1713425378.639382 trunc from 0x034128 to 0x00eb05 (0x25622 bytes) 000045 1713425378.659888 write 0x02960e thru 0x033316 (0x9d09 bytes) 000046 1713425378.662138 MIRROR_EXTEND to 4 mirrors 000049 1713425378.847459 mapread 0x008cc8 thru 0x00a2f6 (0x162f bytes) 000052 1713425378.855592 read 0x02bdbe thru 0x033316 (0x7559 bytes) 000053 1713425378.859315 trunc from 0x033317 to 0x0242af (0xf067 bytes) 000054 1713425378.891158 read 0x00c194 thru 0x00c616 (0x483 bytes) 000055 1713425378.896664 read 0x01c0ea thru 0x0242af (0x81c6 bytes) 000056 1713425378.897087 write 0x008467 thru 0x00f2d5 (0x6e6f bytes) 000058 1713425378.898053 mapread 0x021f13 thru 0x0242af (0x239d bytes) 000061 1713425378.900952 MIRROR_RESYNC 4 mirrors truncating to largest ever: 0x3ce02 000063 1713425379.104614 trunc from 0x0242b0 to 0x03ce01 (0x18b52 bytes) 000064 1713425379.124350 read 0x005f8c thru 0x00f546 (0x95bb bytes) 000066 1713425379.127462 read 0x009653 thru 0x01726e (0xdc1c bytes) 000067 1713425379.133063 trunc from 0x03ce02 to 0x02e977 (0xe48a bytes) 000069 1713425379.140195 mapwrite 0x010fbe thru 0x02002b (0xf06e bytes) 000070 1713425379.167994 read 0x01267e thru 0x014a1e (0x23a1 bytes) 000071 1713425379.168662 mapread 0x01370c thru 0x01a493 (0x6d88 bytes) 000072 1713425379.171569 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000073 1713425379.233939 mapwrite 0x02c02d thru 0x03895a (0xc92e bytes) 000077 1713425379.273948 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000079 1713425379.354411 mapread 0x029437 thru 0x029d53 (0x91d bytes) 000082 1713425379.361180 MIRROR_EXTEND to 3 mirrors 000084 1713425379.563373 mapread 0x002bbe thru 0x005b80 (0x2fc3 bytes) 000085 1713425379.575807 write 0x030974 thru 0x0381c1 (0x784e bytes) 000086 1713425379.602751 read 0x00fcd2 thru 0x013f9f (0x42ce bytes) 000087 1713425379.604499 trunc from 0x03895b to 0x00ce59 (0x2bb01 bytes) 000088 1713425379.611290 mapwrite 0x025ce3 thru 0x02a505 (0x4823 bytes) 000089 1713425379.635881 trunc from 0x02a506 to 0x026e06 (0x36ff bytes) 000091 1713425379.642171 mapwrite 0x03b2ca thru 0x03b6b6 (0x3ed bytes) 000092 1713425379.665384 mapread 0x0285ef thru 0x033645 (0xb057 bytes) 000093 1713425379.671701 write 0x005ce8 thru 0x00e43a (0x8753 bytes) 000096 1713425379.676069 mapwrite 0x01dc5f thru 0x01f75d (0x1aff bytes) 000097 1713425379.701571 MIRROR_EXTEND to 4 mirrors 000099 1713425379.873514 trunc from 0x03b6b7 to 0x027c69 (0x13a4d bytes) 000103 1713425379.909672 trunc from 0x027c6a to 0x00a005 (0x1dc64 bytes) 000104 1713425379.920950 MIRROR_RESYNC 4 mirrors 000106 1713425380.151638 trunc from 0x00a006 to 0x030ca0 (0x26c9b bytes) 000108 1713425380.170577 read 0x01e0c2 thru 0x0286ec (0xa62b bytes) 000110 1713425380.178085 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000112 1713425380.262736 mapread 0x02d83c thru 0x02da1a (0x1df bytes) 000114 1713425380.267397 mapread 0x01f78c thru 0x028817 (0x908c bytes) 000116 1713425380.282049 MIRROR_EXTEND to 4 mirrors 000118 1713425380.483140 trunc from 0x030ca1 to 0x00073b (0x30565 bytes) 000119 1713425380.519395 write 0x02f578 thru 0x03bd84 (0xc80d bytes) 000120 1713425380.521075 mapwrite 0x01b0aa thru 0x0299d7 (0xe92e bytes) 000122 1713425380.569728 read 0x008cbe thru 0x009cbd (0x1000 bytes) 000124 1713425380.573473 write 0x00e6c5 thru 0x016743 (0x807f bytes) 000125 1713425380.578126 mapwrite 0x027ccf thru 0x02d282 (0x55b4 bytes) 000127 1713425380.604251 mapwrite 0x000177 thru 0x00c461 (0xc2eb bytes) 000130 1713425380.640816 mapwrite 0x009fee thru 0x00ad5e (0xd71 bytes) 000131 1713425380.665066 read 0x01622b thru 0x01b09c (0x4e72 bytes) 000132 1713425380.666044 write 0x03b40b thru 0x03ffff (0x4bf5 bytes) 000134 1713425380.668209 mapread 0x0379da thru 0x03ffff (0x8626 bytes) 000138 1713425380.670848 trunc from 0x040000 to 0x026b51 (0x194ae bytes) 000142 1713425380.682811 MIRROR_EXTEND to 5 mirrors 000143 1713425380.877793 mapread 0x008de9 thru 0x015662 (0xc87a bytes) 000144 1713425380.889142 trunc from 0x026b52 to 0x004602 (0x2254f bytes) 000145 1713425380.915527 trunc from 0x004603 to 0x0304fc (0x2befa bytes) 000150 1713425380.923019 mapwrite 0x02a4f8 thru 0x037d12 (0xd81b bytes) 000151 1713425380.957493 mapwrite 0x00e474 thru 0x012166 (0x3cf3 bytes) 000152 1713425380.985160 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000153 1713425381.074374 trunc from 0x037d13 to 0x01ff5e (0x17db4 bytes) 000156 1713425381.083610 MIRROR_EXTEND to 5 mirrors 000159 1713425381.260433 trunc from 0x01ff5f to 0x01c396 (0x3bc8 bytes) 000160 1713425381.291287 trunc from 0x01c397 to 0x02273c (0x63a6 bytes) 000161 1713425381.300691 read 0x001a43 thru 0x0073a9 (0x5967 bytes) 000162 1713425381.305706 trunc from 0x02273d to 0x017a51 (0xaceb bytes) 000164 1713425381.313254 write 0x030f4e thru 0x03dcfe (0xcdb1 bytes) 000167 1713425381.315869 write 0x02beb9 thru 0x03b7f8 (0xf940 bytes) 000168 1713425381.318810 mapread 0x01b8fe thru 0x02200f (0x6712 bytes) 000169 1713425381.324815 read 0x00e05e thru 0x012c24 (0x4bc7 bytes) 000170 1713425381.328696 write 0x00f251 thru 0x01c264 (0xd014 bytes) 000172 1713425381.329909 trunc from 0x03dcff to 0x036f88 (0x6d76 bytes) 000174 1713425381.335358 mapwrite 0x00529a thru 0x00c8fc (0x7663 bytes) 000175 1713425381.356418 write 0x007b20 thru 0x007d60 (0x241 bytes) 000181 1713425381.360563 MIRROR_RESYNC 5 mirrors 000182 1713425381.557793 mapread 0x00ddb4 thru 0x01b489 (0xd6d6 bytes) 000185 1713425381.566816 mapwrite 0x01e4c5 thru 0x0266d9 (0x8215 bytes) 000187 1713425381.634028 write 0x01d118 thru 0x0237c8 (0x66b1 bytes) 000188 1713425381.635068 mapwrite 0x03d6f5 thru 0x03ffff (0x290b bytes) 000189 1713425381.681427 trunc from 0x040000 to 0x0027f4 (0x3d80b bytes) 000192 1713425381.693025 MIRROR_EXTEND to 6 mirrors 000194 1713425381.872898 mapread 0x001f3b thru 0x0027f4 (0x8ba bytes) 000195 1713425381.876799 trunc from 0x0027f5 to 0x0170d5 (0x148e1 bytes) 000196 1713425381.906925 read 0x012c8c thru 0x0170d5 (0x444a bytes) 000197 1713425381.911841 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000202 1713425381.988556 mapwrite 0x03eb4a thru 0x03ffff (0x14b6 bytes) 000203 1713425382.027611 trunc from 0x040000 to 0x00c133 (0x33ecc bytes) 000206 1713425382.038207 read 0x00b423 thru 0x00c133 (0xd11 bytes) 000209 1713425382.043904 read 0x0049a9 thru 0x007922 (0x2f7a bytes) 000211 1713425382.053761 trunc from 0x00c134 to 0x02f4fb (0x233c8 bytes) 000212 1713425382.060884 read 0x01f880 thru 0x02c787 (0xcf08 bytes) 000213 1713425382.067122 read 0x00f6c4 thru 0x010d59 (0x1696 bytes) 000214 1713425382.067601 read 0x01bb2a thru 0x022b0b (0x6fe2 bytes) 000217 1713425382.068720 mapwrite 0x034df4 thru 0x03a727 (0x5934 bytes) 000221 1713425382.114475 MIRROR_RESYNC 5 mirrors 000222 1713425382.358103 mapread 0x01b368 thru 0x0279fa (0xc693 bytes) 000224 1713425382.366113 trunc from 0x03a728 to 0x03580b (0x4f1c bytes) 000225 1713425382.390912 read 0x00b8b5 thru 0x019805 (0xdf51 bytes) 000226 1713425382.394015 write 0x01e696 thru 0x02c092 (0xd9fd bytes) 000229 1713425382.398779 read 0x02f55c thru 0x03580b (0x62b0 bytes) 000230 1713425382.401636 write 0x02aa8b thru 0x02f1d4 (0x474a bytes) 000231 1713425382.402134 write 0x0020b1 thru 0x0078b2 (0x5802 bytes) 000232 1713425382.402708 mapread 0x01a702 thru 0x02278f (0x808e bytes) 000233 1713425382.403156 mapread 0x0081f0 thru 0x00ecbb (0x6acc bytes) 000235 1713425382.403858 mapread 0x03248f thru 0x03580b (0x337d bytes) 000236 1713425382.404226 read 0x02b9e6 thru 0x02cdee (0x1409 bytes) 000237 1713425382.404492 mapwrite 0x02dbff thru 0x03005b (0x245d bytes) 000239 1713425382.421198 trunc from 0x03580c to 0x00f6ff (0x2610c bytes) 000241 1713425382.427166 mapread 0x00f1c5 thru 0x00f6ff (0x53b bytes) 000244 1713425382.428407 write 0x01f352 thru 0x02f164 (0xfe13 bytes) 000245 1713425382.430727 MIRROR_EXTEND to 6 mirrors 000250 1713425382.550753 trunc from 0x02f165 to 0x02932d (0x5e37 bytes) 000251 1713425382.567707 mapwrite 0x021c92 thru 0x02bee7 (0xa256 bytes) 000252 1713425382.591093 read 0x011530 thru 0x013f33 (0x2a04 bytes) truncating to largest ever: 0x3d609 000254 1713425382.593972 trunc from 0x02bee8 to 0x03d608 (0x11721 bytes) 000255 1713425382.597531 trunc from 0x03d609 to 0x000446 (0x3d1c2 bytes) 000256 1713425382.602824 MIRROR_RESYNC 6 mirrors 000260 1713425382.747567 read 0x0000f4 thru 0x000446 (0x353 bytes) 000261 1713425382.747803 mapread 0x000325 thru 0x000446 (0x122 bytes) 000263 1713425382.748246 mapread 0x000339 thru 0x000446 (0x10e bytes) 000266 1713425382.748663 read 0x000378 thru 0x000446 (0xcf bytes) 000268 1713425382.748919 write 0x02130b thru 0x02d46e (0xc164 bytes) 000269 1713425382.762020 read 0x015411 thru 0x01a675 (0x5265 bytes) 000274 1713425382.767166 MIRROR_EXTEND to 7 mirrors 000275 1713425382.868856 mapwrite 0x00a014 thru 0x00db83 (0x3b70 bytes) 000277 1713425382.913300 write 0x01f3c4 thru 0x02ea24 (0xf661 bytes) 000278 1713425382.916536 read 0x00db91 thru 0x015d36 (0x81a6 bytes) 000279 1713425382.918056 mapread 0x0167ca thru 0x01c013 (0x584a bytes) 000280 1713425382.919694 mapread 0x029f49 thru 0x02cf6c (0x3024 bytes) 000284 1713425382.920735 read 0x026927 thru 0x029305 (0x29df bytes) 000286 1713425382.922572 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000287 1713425382.967844 mapwrite 0x03c300 thru 0x03ffff (0x3d00 bytes) 000294 1713425382.994892 trunc from 0x040000 to 0x03cd05 (0x32fa bytes) 000296 1713425383.000171 MIRROR_EXTEND to 7 mirrors 000297 1713425383.114003 mapread 0x006322 thru 0x00f288 (0x8f67 bytes) 000299 1713425383.124421 read 0x028527 thru 0x03190c (0x93e6 bytes) 000300 1713425383.128533 read 0x03c1a9 thru 0x03cd05 (0xb5d bytes) 000301 1713425383.130606 trunc from 0x03cd06 to 0x003669 (0x3969c bytes) 000302 1713425383.146866 trunc from 0x00366a to 0x014ebf (0x11856 bytes) 000303 1713425383.151478 trunc from 0x014ec0 to 0x01ca91 (0x7bd2 bytes) 000304 1713425383.155647 write 0x03f108 thru 0x03ffff (0xef8 bytes) 000305 1713425383.156276 mapwrite 0x0089ad thru 0x0144f8 (0xbb4c bytes) 000306 1713425383.178460 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000307 1713425383.235920 write 0x023a77 thru 0x0310ea (0xd674 bytes) 000308 1713425383.240705 mapread 0x03ee50 thru 0x03ffff (0x11b0 bytes) 000310 1713425383.244854 mapread 0x010563 thru 0x018b81 (0x861f bytes) 000315 1713425383.254077 mapread 0x00f6f0 thru 0x01ecef (0xf600 bytes) 000316 1713425383.255074 write 0x027505 thru 0x02755b (0x57 bytes) 000317 1713425383.255575 write 0x028768 thru 0x030fdc (0x8875 bytes) 000318 1713425383.258826 MIRROR_RESYNC 6 mirrors 000319 1713425383.505960 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000320 1713425383.627883 mapwrite 0x00f4c0 thru 0x01b3e4 (0xbf25 bytes) 000321 1713425383.710183 trunc from 0x040000 to 0x0250d0 (0x1af2f bytes) 000322 1713425383.726659 read 0x019e85 thru 0x0250d0 (0xb24c bytes) 000323 1713425383.732267 MIRROR_EXTEND to 6 mirrors 000324 1713425383.939364 MIRROR_RESYNC 6 mirrors 000325 1713425384.248452 write 0x0125c6 thru 0x015728 (0x3163 bytes) 000326 1713425384.278039 mapwrite 0x036274 thru 0x03b17f (0x4f0c bytes) 000329 1713425384.328740 mapwrite 0x0085d6 thru 0x016cc6 (0xe6f1 bytes) 000330 1713425384.365676 read 0x02393b thru 0x025adc (0x21a2 bytes) 000331 1713425384.369226 trunc from 0x03b180 to 0x0316cd (0x9ab2 bytes) 000332 1713425384.377722 trunc from 0x0316ce to 0x0039ec (0x2dce1 bytes) 000333 1713425384.385148 mapread 0x002872 thru 0x0039ec (0x117b bytes) 000335 1713425384.386317 write 0x006ee7 thru 0x010c83 (0x9d9d bytes) 000336 1713425384.389593 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000337 1713425384.463224 mapwrite 0x01f99e thru 0x02e586 (0xebe9 bytes) 000338 1713425384.515376 MIRROR_EXTEND to 6 mirrors 000339 1713425384.714266 MIRROR_EXTEND to 7 mirrors 000340 1713425384.870226 MIRROR_RESYNC 7 mirrors 000341 1713425385.077275 write 0x01fad5 thru 0x02bbbd (0xc0e9 bytes) 000342 1713425385.107462 mapread 0x0059fd thru 0x01167d (0xbc81 bytes) 000343 1713425385.117056 write 0x004e78 thru 0x006b1b (0x1ca4 bytes) 000344 1713425385.117961 mapread 0x010277 thru 0x01d682 (0xd40c bytes) 000349 1713425385.124819 mapwrite 0x012c56 thru 0x018061 (0x540c bytes) 000350 1713425385.158656 read 0x025c98 thru 0x02e586 (0x88ef bytes) 000352 1713425385.165166 MIRROR_RESYNC 7 mirrors 000353 1713425385.487061 mapwrite 0x01acc8 thru 0x01ba88 (0xdc1 bytes) 000354 1713425385.543176 MIRROR_EXTEND to 8 mirrors 000356 1713425385.673362 trunc from 0x02e587 to 0x02a13c (0x444a bytes) 000358 1713425385.697378 mapwrite 0x02ca9d thru 0x0392b2 (0xc816 bytes) 000361 1713425385.736233 MIRROR_RESYNC 8 mirrors 000362 1713425386.053297 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000363 1713425386.184188 trunc from 0x0392b3 to 0x0376b9 (0x1bf9 bytes) 000364 1713425386.222169 write 0x03318c thru 0x037c4a (0x4abf bytes) 000365 1713425386.227904 write 0x02da6f thru 0x03a9d8 (0xcf6a bytes) 000367 1713425386.233758 write 0x00b8f7 thru 0x0105d8 (0x4ce2 bytes) 000368 1713425386.238441 read 0x036c07 thru 0x03a9d8 (0x3dd2 bytes) 000370 1713425386.239430 mapwrite 0x02d840 thru 0x032987 (0x5148 bytes) 000372 1713425386.272108 trunc from 0x03a9d9 to 0x021500 (0x194d8 bytes) 000374 1713425386.282841 read 0x0214bd thru 0x021500 (0x44 bytes) 000375 1713425386.286970 write 0x03a276 thru 0x03d26a (0x2ff5 bytes) 000376 1713425386.290118 MIRROR_EXTEND to 8 mirrors 000377 1713425386.476845 mapwrite 0x01ee7a thru 0x029c14 (0xad9b bytes) 000378 1713425386.561195 mapread 0x018801 thru 0x01b719 (0x2f19 bytes) 000379 1713425386.565998 mapread 0x01fae5 thru 0x026ffc (0x7518 bytes) 000380 1713425386.566851 mapread 0x00df10 thru 0x017596 (0x9687 bytes) 000381 1713425386.570902 trunc from 0x03d26b to 0x006302 (0x36f68 bytes) 000383 1713425386.589391 read 0x005a8e thru 0x006302 (0x875 bytes) 000384 1713425386.592063 trunc from 0x006303 to 0x03755f (0x3125d bytes) 000386 1713425386.599536 mapwrite 0x0369f2 thru 0x03a169 (0x3778 bytes) 000387 1713425386.634179 read 0x000d0c thru 0x00ccc9 (0xbfbe bytes) 000390 1713425386.642541 write 0x016368 thru 0x02527c (0xef15 bytes) 000395 1713425386.646135 write 0x004271 thru 0x01395b (0xf6eb bytes) 000396 1713425386.647277 trunc from 0x03a16a to 0x00b567 (0x2ec02 bytes) 000398 1713425386.655310 mapwrite 0x033871 thru 0x038273 (0x4a03 bytes) 000399 1713425386.697347 trunc from 0x038274 to 0x00709d (0x311d6 bytes) 000400 1713425386.703982 mapread 0x004646 thru 0x00709d (0x2a58 bytes) 000401 1713425386.706579 MIRROR_EXTEND to 9 mirrors 000402 1713425386.865044 read 0x005eb3 thru 0x00709d (0x11eb bytes) 000403 1713425386.869236 mapwrite 0x02f23a thru 0x03d185 (0xdf4c bytes) 000404 1713425386.929867 trunc from 0x03d186 to 0x01c202 (0x20f83 bytes) 000405 1713425386.938912 read 0x00d7b5 thru 0x010242 (0x2a8e bytes) 000406 1713425386.945392 mapwrite 0x00bae4 thru 0x0183a9 (0xc8c6 bytes) 000408 1713425386.977754 MIRROR_EXTEND to 10 mirrors 000409 1713425387.157936 read 0x00f5b0 thru 0x010c86 (0x16d7 bytes) 000411 1713425387.164060 mapread 0x00de38 thru 0x00fb35 (0x1cfe bytes) 000413 1713425387.173900 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000414 1713425387.252154 mapread 0x001fd1 thru 0x0071ae (0x51de bytes) 000415 1713425387.263560 read 0x008ba7 thru 0x00df68 (0x53c2 bytes) 000419 1713425387.280391 write 0x03cfc5 thru 0x03ffff (0x303b bytes) 000420 1713425387.296050 mapread 0x02c5c6 thru 0x02e04f (0x1a8a bytes) 000423 1713425387.299757 mapread 0x0212fa thru 0x03048a (0xf191 bytes) 000424 1713425387.302523 write 0x03ac5b thru 0x03ffff (0x53a5 bytes) 000425 1713425387.305082 trunc from 0x040000 to 0x00f731 (0x308ce bytes) 000428 1713425387.313458 MIRROR_RESYNC 9 mirrors 000429 1713425387.565725 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000431 1713425387.637198 trunc from 0x00f732 to 0x032a69 (0x23338 bytes) 000432 1713425387.652880 write 0x01bda0 thru 0x021be5 (0x5e46 bytes) 000434 1713425387.655702 trunc from 0x032a6a to 0x01e137 (0x14932 bytes) 000436 1713425387.660998 mapwrite 0x02e02a thru 0x03bb3e (0xdb15 bytes) 000438 1713425387.688141 mapread 0x008d06 thru 0x00af7d (0x2278 bytes) 000439 1713425387.691784 mapread 0x01d93f thru 0x02a057 (0xc719 bytes) 000440 1713425387.694548 mapread 0x00644c thru 0x00f721 (0x92d6 bytes) 000441 1713425387.695030 mapread 0x02c80b thru 0x03488f (0x8085 bytes) 000442 1713425387.695493 trunc from 0x03bb3f to 0x0189ea (0x23154 bytes) 000443 1713425387.700003 read 0x000489 thru 0x00f6c1 (0xf239 bytes) 000444 1713425387.702677 trunc from 0x0189eb to 0x01d2f2 (0x4908 bytes) 000445 1713425387.706842 trunc from 0x01d2f3 to 0x00be1d (0x114d5 bytes) 000447 1713425387.713581 MIRROR_RESYNC 8 mirrors 000448 1713425387.925667 write 0x02f709 thru 0x038ea3 (0x979b bytes) 000449 1713425387.944213 read 0x02b68b thru 0x033ad4 (0x844a bytes) 000451 1713425387.947007 write 0x026de8 thru 0x0357bc (0xe9d5 bytes) 000452 1713425387.948921 trunc from 0x038ea4 to 0x021b16 (0x1738d bytes) 000453 1713425387.957606 write 0x019e1b thru 0x024d59 (0xaf3f bytes) 000455 1713425387.961711 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000458 1713425388.025501 trunc from 0x024d5a to 0x02b1f9 (0x64a0 bytes) 000461 1713425388.033066 write 0x034006 thru 0x038831 (0x482c bytes) 000463 1713425388.035233 mapread 0x00924d thru 0x00d84a (0x45fe bytes) 000464 1713425388.042861 trunc from 0x038832 to 0x037610 (0x1221 bytes) 000466 1713425388.049745 MIRROR_EXTEND to 8 mirrors 000467 1713425388.211206 mapwrite 0x00459c thru 0x01219f (0xdc04 bytes) 000469 1713425388.285488 write 0x00d44a thru 0x013a93 (0x664a bytes) 000471 1713425388.286761 read 0x009b39 thru 0x0157ab (0xbc73 bytes) 000473 1713425388.289975 MIRROR_RESYNC 8 mirrors 000477 1713425388.586751 write 0x03d8fb thru 0x03ffff (0x2705 bytes) 000478 1713425388.601426 trunc from 0x040000 to 0x00f655 (0x309aa bytes) 000481 1713425388.611645 MIRROR_EXTEND to 9 mirrors 000482 1713425388.714788 mapread 0x002aa5 thru 0x002d6b (0x2c7 bytes) 000484 1713425388.719972 read 0x00aae7 thru 0x00f655 (0x4b6f bytes) 000485 1713425388.723786 write 0x013245 thru 0x01b5ee (0x83aa bytes) 000486 1713425388.742670 write 0x037d80 thru 0x03c04f (0x42d0 bytes) 000487 1713425388.743612 mapread 0x02e921 thru 0x02f77d (0xe5d bytes) 000488 1713425388.748134 write 0x01dd1c thru 0x02522d (0x7512 bytes) 000489 1713425388.753025 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000490 1713425388.818731 mapwrite 0x017bc5 thru 0x019822 (0x1c5e bytes) 000491 1713425388.844018 trunc from 0x03c050 to 0x011b67 (0x2a4e8 bytes) 000493 1713425388.854951 mapread 0x00a6cb thru 0x011b67 (0x749d bytes) 000494 1713425388.858057 mapwrite 0x0100c8 thru 0x0106c4 (0x5fd bytes) 000496 1713425388.874158 mapwrite 0x0179ad thru 0x0192d3 (0x1927 bytes) 000497 1713425388.897934 write 0x036e69 thru 0x03ffff (0x9197 bytes) 000500 1713425388.899823 write 0x00e88e thru 0x00ec84 (0x3f7 bytes) 000501 1713425388.900593 trunc from 0x040000 to 0x0352af (0xad50 bytes) 000502 1713425388.907784 MIRROR_RESYNC 8 mirrors 000504 1713425389.173679 trunc from 0x0352b0 to 0x01cb87 (0x18728 bytes) 000506 1713425389.196326 trunc from 0x01cb88 to 0x03cc46 (0x200bf bytes) 000507 1713425389.203375 mapwrite 0x009534 thru 0x00af69 (0x1a36 bytes) 000509 1713425389.227955 mapwrite 0x0190de thru 0x02649f (0xd3c2 bytes) 000511 1713425389.253736 mapread 0x029303 thru 0x02aa46 (0x1744 bytes) 000514 1713425389.254900 write 0x004058 thru 0x007158 (0x3101 bytes) 000516 1713425389.258592 mapread 0x00ff8e thru 0x01e5b6 (0xe629 bytes) 000517 1713425389.262967 MIRROR_RESYNC 8 mirrors 000518 1713425389.558786 write 0x0040f6 thru 0x012e55 (0xed60 bytes) 000520 1713425389.590252 write 0x01b75d thru 0x026e55 (0xb6f9 bytes) 000521 1713425389.597724 mapread 0x011f4d thru 0x01a0cf (0x8183 bytes) 000522 1713425389.600824 mapread 0x00a066 thru 0x00fefc (0x5e97 bytes) 000523 1713425389.601659 trunc from 0x03cc47 to 0x0039fb (0x3924b bytes) 000524 1713425389.617445 read 0x002c72 thru 0x0039fb (0xd8a bytes) 000526 1713425389.620913 read 0x002353 thru 0x0039fb (0x16a9 bytes) 000527 1713425389.621468 read 0x00325b thru 0x0039fb (0x7a1 bytes) 000528 1713425389.622107 mapwrite 0x015128 thru 0x019cea (0x4bc3 bytes) 000530 1713425389.657483 mapwrite 0x020122 thru 0x022b4f (0x2a2e bytes) 000531 1713425389.683577 write 0x0382e1 thru 0x03844b (0x16b bytes) 000532 1713425389.684217 read 0x02d9c8 thru 0x03844b (0xaa84 bytes) 000534 1713425389.687954 mapwrite 0x00284d thru 0x006909 (0x40bd bytes) 000535 1713425389.708954 write 0x036e30 thru 0x03ffff (0x91d0 bytes) 000536 1713425389.709905 write 0x033d52 thru 0x038cce (0x4f7d bytes) 000537 1713425389.710686 read 0x037f8b thru 0x03b0c6 (0x313c bytes) 000538 1713425389.711045 mapread 0x027243 thru 0x028854 (0x1612 bytes) 000541 1713425389.712180 mapwrite 0x009a32 thru 0x011e06 (0x83d5 bytes) 000544 1713425389.736935 mapread 0x018dd8 thru 0x01fac6 (0x6cef bytes) 000545 1713425389.737573 mapwrite 0x0190cd thru 0x025cb2 (0xcbe6 bytes) 000546 1713425389.761837 mapread 0x0072b6 thru 0x00881e (0x1569 bytes) 000547 1713425389.762673 read 0x015999 thru 0x0247bf (0xee27 bytes) 000551 1713425389.765639 mapwrite 0x03200f thru 0x03fc86 (0xdc78 bytes) 000552 1713425389.794202 trunc from 0x040000 to 0x03551b (0xaae4 bytes) 000555 1713425389.802202 read 0x00f27c thru 0x010fd0 (0x1d55 bytes) 000557 1713425389.803867 read 0x00f653 thru 0x010de3 (0x1791 bytes) 000559 1713425389.808121 MIRROR_RESYNC 8 mirrors 000562 1713425390.232840 mapread 0x00a572 thru 0x00bcb3 (0x1742 bytes) 000565 1713425390.238484 write 0x0116b2 thru 0x01431a (0x2c69 bytes) 000567 1713425390.267589 read 0x016228 thru 0x01bd82 (0x5b5b bytes) 000569 1713425390.272139 mapread 0x00a9b7 thru 0x00e2f6 (0x3940 bytes) 000570 1713425390.272895 mapwrite 0x013020 thru 0x018253 (0x5234 bytes) 000572 1713425390.302061 write 0x013eab thru 0x021e38 (0xdf8e bytes) 000574 1713425390.306110 mapwrite 0x0205eb thru 0x022c9e (0x26b4 bytes) 000575 1713425390.337239 write 0x03da0f thru 0x03ffff (0x25f1 bytes) 000576 1713425390.338362 mapwrite 0x019ee0 thru 0x01dd3f (0x3e60 bytes) 000577 1713425390.366773 trunc from 0x040000 to 0x01450d (0x2baf2 bytes) 000579 1713425390.380992 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000581 1713425390.472583 read 0x01180e thru 0x01450d (0x2d00 bytes) 000582 1713425390.479555 MIRROR_RESYNC 7 mirrors 000583 1713425390.825499 mapread 0x00cf85 thru 0x01450d (0x7589 bytes) 000584 1713425390.835069 mapwrite 0x036e0c thru 0x03dff1 (0x71e6 bytes) 000585 1713425390.917151 mapread 0x019605 thru 0x028467 (0xee63 bytes) 000586 1713425390.926893 MIRROR_EXTEND to 8 mirrors 000587 1713425391.123356 mapwrite 0x00bd4e thru 0x01890f (0xcbc2 bytes) 000588 1713425391.180127 mapread 0x03a960 thru 0x03dff1 (0x3692 bytes) 000589 1713425391.184309 mapwrite 0x01aa56 thru 0x021a75 (0x7020 bytes) 000594 1713425391.208045 mapread 0x02e332 thru 0x03163d (0x330c bytes) 000595 1713425391.213095 mapread 0x036358 thru 0x038508 (0x21b1 bytes) 000597 1713425391.214241 mapwrite 0x013a7d thru 0x01ec7f (0xb203 bytes) 000598 1713425391.236170 mapread 0x015072 thru 0x022624 (0xd5b3 bytes) 000599 1713425391.237037 read 0x02aee1 thru 0x039fcb (0xf0eb bytes) 000601 1713425391.238090 mapread 0x01d844 thru 0x01d9b0 (0x16d bytes) 000602 1713425391.238818 trunc from 0x03dff2 to 0x013a40 (0x2a5b1 bytes) 000603 1713425391.248324 mapwrite 0x035b9f thru 0x03953f (0x39a1 bytes) 000604 1713425391.287719 mapread 0x003ac9 thru 0x008c5e (0x5196 bytes) 000607 1713425391.292099 read 0x015824 thru 0x0211f6 (0xb9d3 bytes) 000608 1713425391.297467 mapwrite 0x03a9f0 thru 0x03ffff (0x5610 bytes) 000611 1713425391.331893 mapwrite 0x009595 thru 0x009947 (0x3b3 bytes) 000612 1713425391.345237 read 0x00c31a thru 0x0120d8 (0x5dbf bytes) 000613 1713425391.345684 mapread 0x0046fe thru 0x00aa2b (0x632e bytes) 000617 1713425391.347221 read 0x038463 thru 0x03c06d (0x3c0b bytes) 000618 1713425391.347687 mapread 0x01c13f thru 0x022f97 (0x6e59 bytes) 000619 1713425391.348331 write 0x032ffb thru 0x033b53 (0xb59 bytes) 000623 1713425391.352510 MIRROR_RESYNC 8 mirrors 000626 1713425391.587531 read 0x00cd5e thru 0x018e54 (0xc0f7 bytes) 000628 1713425391.589747 trunc from 0x040000 to 0x02fba6 (0x10459 bytes) 000629 1713425391.606404 trunc from 0x02fba7 to 0x00e2d0 (0x218d6 bytes) 000630 1713425391.610811 read 0x00681d thru 0x007573 (0xd57 bytes) 000632 1713425391.614255 read 0x003a19 thru 0x00c1b3 (0x879b bytes) 000635 1713425391.615488 write 0x00acef thru 0x00eefa (0x420c bytes) 000636 1713425391.616136 write 0x00d5f9 thru 0x016016 (0x8a1e bytes) 000637 1713425391.616842 mapread 0x003a12 thru 0x00bc09 (0x81f8 bytes) 000638 1713425391.617461 trunc from 0x016017 to 0x01dd8b (0x7d75 bytes) 000640 1713425391.621741 mapread 0x0085ce thru 0x00ba5d (0x3490 bytes) 000641 1713425391.622390 mapread 0x008246 thru 0x00b444 (0x31ff bytes) 000644 1713425391.623533 mapwrite 0x013217 thru 0x016e25 (0x3c0f bytes) 000645 1713425391.640831 mapwrite 0x03f421 thru 0x03ffff (0xbdf bytes) 000647 1713425391.665495 mapread 0x00b831 thru 0x018f51 (0xd721 bytes) 000648 1713425391.666017 write 0x0334a9 thru 0x0390e6 (0x5c3e bytes) 000649 1713425391.666665 write 0x02fc71 thru 0x037061 (0x73f1 bytes) 000650 1713425391.667339 read 0x03965f thru 0x03ffff (0x69a1 bytes) 000652 1713425391.667876 write 0x00c341 thru 0x018d3d (0xc9fd bytes) 000653 1713425391.668791 trunc from 0x040000 to 0x01e23b (0x21dc4 bytes) 000654 1713425391.675901 MIRROR_RESYNC 8 mirrors 000655 1713425391.928432 trunc from 0x01e23c to 0x030b85 (0x1294a bytes) 000660 1713425391.947103 mapwrite 0x0258fd thru 0x02d2ac (0x79b0 bytes) 000661 1713425391.970126 mapread 0x00e436 thru 0x01275f (0x432a bytes) 000663 1713425391.973250 mapwrite 0x0057c8 thru 0x00785c (0x2095 bytes) 000668 1713425391.992060 MIRROR_EXTEND to 9 mirrors 000669 1713425392.153953 write 0x0184f1 thru 0x028160 (0xfc70 bytes) 000670 1713425392.171939 trunc from 0x030b86 to 0x028e23 (0x7d62 bytes) 000671 1713425392.177654 read 0x0231ed thru 0x028e23 (0x5c37 bytes) 000672 1713425392.178119 read 0x013d9b thru 0x01a33a (0x65a0 bytes) 000673 1713425392.183125 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000674 1713425392.248255 trunc from 0x028e24 to 0x0273d0 (0x1a53 bytes) 000675 1713425392.256707 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000676 1713425392.311556 mapread 0x01cca5 thru 0x0273d0 (0xa72c bytes) 000678 1713425392.324811 mapread 0x00943e thru 0x00a1f4 (0xdb7 bytes) 000679 1713425392.329724 trunc from 0x0273d1 to 0x007089 (0x20347 bytes) 000681 1713425392.338146 write 0x039d9b thru 0x03ffff (0x6265 bytes) 000682 1713425392.339657 read 0x0049ce thru 0x0070f7 (0x272a bytes) 000685 1713425392.344087 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000686 1713425392.435978 read 0x02c658 thru 0x03454e (0x7ef7 bytes) 000687 1713425392.441291 mapwrite 0x01482e thru 0x018cc2 (0x4495 bytes) 000694 1713425392.480009 read 0x005cab thru 0x009cfe (0x4054 bytes) 000699 1713425392.492063 MIRROR_EXTEND to 7 mirrors 000700 1713425392.671912 write 0x026646 thru 0x0316c1 (0xb07c bytes) 000701 1713425392.703311 mapwrite 0x02412e thru 0x033f09 (0xfddc bytes) 000702 1713425392.740790 MIRROR_EXTEND to 8 mirrors 000703 1713425392.904478 read 0x0103f7 thru 0x0146bc (0x42c6 bytes) 000704 1713425392.907479 mapread 0x008085 thru 0x00a9c1 (0x293d bytes) 000706 1713425392.915006 mapread 0x01794d thru 0x019f69 (0x261d bytes) 000707 1713425392.920997 MIRROR_EXTEND to 9 mirrors 000709 1713425393.056372 mapread 0x03a18d thru 0x03ffff (0x5e73 bytes) 000710 1713425393.065237 read 0x007479 thru 0x00b567 (0x40ef bytes) 000712 1713425393.071342 mapread 0x02d57a thru 0x0358fd (0x8384 bytes) 000715 1713425393.082037 read 0x019e2a thru 0x0206cf (0x68a6 bytes) 000717 1713425393.090631 read 0x03b20a thru 0x03ffff (0x4df6 bytes) 000719 1713425393.094809 trunc from 0x040000 to 0x02d610 (0x129ef bytes) 000720 1713425393.133541 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000723 1713425393.213867 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000724 1713425393.301968 mapread 0x025e89 thru 0x02a574 (0x46ec bytes) 000725 1713425393.315205 MIRROR_EXTEND to 8 mirrors 000726 1713425393.496813 read 0x01a974 thru 0x021861 (0x6eee bytes) 000727 1713425393.502754 mapwrite 0x036be7 thru 0x03ffff (0x9419 bytes) 000728 1713425393.553015 mapread 0x0250c3 thru 0x031e65 (0xcda3 bytes) 000729 1713425393.558512 read 0x0344d9 thru 0x03af31 (0x6a59 bytes) 000730 1713425393.558831 mapwrite 0x01e70e thru 0x02c778 (0xe06b bytes) 000731 1713425393.581014 mapread 0x01eb36 thru 0x024a3d (0x5f08 bytes) 000733 1713425393.582167 read 0x017590 thru 0x024b57 (0xd5c8 bytes) 000734 1713425393.582665 trunc from 0x040000 to 0x00433b (0x3bcc4 bytes) truncating to largest ever: 0x3e124 000736 1713425393.590984 trunc from 0x00433c to 0x03e123 (0x39de8 bytes) 000737 1713425393.596509 write 0x014d44 thru 0x0242e9 (0xf5a6 bytes) 000738 1713425393.603787 mapread 0x025459 thru 0x026ad5 (0x167d bytes) 000739 1713425393.604369 mapread 0x032d9f thru 0x03e123 (0xb385 bytes) 000740 1713425393.607865 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000741 1713425393.694576 read 0x032730 thru 0x032ab7 (0x388 bytes) 000747 1713425393.700275 mapwrite 0x030e19 thru 0x035d56 (0x4f3e bytes) 000748 1713425393.740780 read 0x02c17f thru 0x034938 (0x87ba bytes) 000749 1713425393.741853 trunc from 0x03e124 to 0x0349ed (0x9736 bytes) 000750 1713425393.747851 trunc from 0x0349ee to 0x015062 (0x1f98b bytes) 000752 1713425393.757719 MIRROR_RESYNC 7 mirrors 000754 1713425394.037219 read 0x003ea3 thru 0x008e5d (0x4fbb bytes) 000757 1713425394.041651 mapread 0x00b712 thru 0x011bf8 (0x64e7 bytes) 000758 1713425394.050675 mapread 0x00eab9 thru 0x015062 (0x65aa bytes) 000759 1713425394.051611 read 0x00cbf6 thru 0x014f29 (0x8334 bytes) 000762 1713425394.052894 mapwrite 0x02be75 thru 0x033e19 (0x7fa5 bytes) 000764 1713425394.124932 mapwrite 0x0120ec thru 0x0183f5 (0x630a bytes) 000766 1713425394.151801 read 0x0285ce thru 0x02e752 (0x6185 bytes) 000767 1713425394.152380 mapread 0x0192e1 thru 0x019458 (0x178 bytes) 000769 1713425394.153671 write 0x0081fb thru 0x0144f4 (0xc2fa bytes) 000770 1713425394.154831 write 0x00b625 thru 0x00cd7e (0x175a bytes) 000771 1713425394.155764 read 0x031efa thru 0x033e19 (0x1f20 bytes) 000774 1713425394.156913 read 0x0087d7 thru 0x018665 (0xfe8f bytes) 000775 1713425394.159684 MIRROR_EXTEND to 8 mirrors 000776 1713425394.382612 mapread 0x026869 thru 0x033e19 (0xd5b1 bytes) 000777 1713425394.393491 mapread 0x01ef43 thru 0x0299ef (0xaaad bytes) 000779 1713425394.401358 trunc from 0x033e1a to 0x002703 (0x31716 bytes) 000780 1713425394.436812 MIRROR_RESYNC 8 mirrors 000781 1713425394.665559 MIRROR_RESYNC 8 mirrors 000782 1713425394.712187 mapwrite 0x01f702 thru 0x028c06 (0x9505 bytes) 000783 1713425394.763856 read 0x00a91e thru 0x017316 (0xc9f9 bytes) 000784 1713425394.767396 read 0x022964 thru 0x028c06 (0x62a3 bytes) 000786 1713425394.769599 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000787 1713425394.819755 MIRROR_RESYNC 7 mirrors 000789 1713425395.119261 mapwrite 0x0356fb thru 0x03ffff (0xa905 bytes) 000790 1713425395.178653 mapwrite 0x03071d thru 0x036207 (0x5aeb bytes) 000792 1713425395.200794 write 0x028cb5 thru 0x0292a5 (0x5f1 bytes) 000793 1713425395.204980 read 0x00cf5c thru 0x010b84 (0x3c29 bytes) 000797 1713425395.209308 mapread 0x033088 thru 0x03f50c (0xc485 bytes) 000799 1713425395.210262 mapwrite 0x023060 thru 0x029b30 (0x6ad1 bytes) 000800 1713425395.232841 write 0x00e130 thru 0x0142f4 (0x61c5 bytes) 000801 1713425395.235825 read 0x0014eb thru 0x005c17 (0x472d bytes) 000802 1713425395.241120 mapwrite 0x03ee27 thru 0x03ffff (0x11d9 bytes) 000803 1713425395.265711 write 0x02c4c8 thru 0x038565 (0xc09e bytes) 000804 1713425395.266810 write 0x03b0f6 thru 0x03f182 (0x408d bytes) 000805 1713425395.267588 write 0x03a0fc thru 0x03d505 (0x340a bytes) 000806 1713425395.268423 trunc from 0x040000 to 0x017c9e (0x28361 bytes) 000807 1713425395.278116 mapread 0x003692 thru 0x00fd8a (0xc6f9 bytes) 000810 1713425395.279653 write 0x034437 thru 0x03c5ed (0x81b7 bytes) 000812 1713425395.281199 read 0x02fa6d thru 0x032dee (0x3382 bytes) 000813 1713425395.287638 mapwrite 0x01b292 thru 0x01ec73 (0x39e2 bytes) 000815 1713425395.316632 mapread 0x03a585 thru 0x03c5ed (0x2069 bytes) 000817 1713425395.320175 MIRROR_RESYNC 7 mirrors 000818 1713425395.666372 write 0x01cf01 thru 0x027469 (0xa569 bytes) 000819 1713425395.697311 read 0x0011d3 thru 0x001529 (0x357 bytes) 000820 1713425395.699857 mapwrite 0x01edaa thru 0x027bb2 (0x8e09 bytes) 000821 1713425395.728969 write 0x003e33 thru 0x0054f4 (0x16c2 bytes) 000822 1713425395.734439 read 0x0208dc thru 0x02dc45 (0xd36a bytes) 000827 1713425395.746689 MIRROR_RESYNC 7 mirrors 000829 1713425396.089822 trunc from 0x03c5ee to 0x02c055 (0x10598 bytes) 000831 1713425396.117990 MIRROR_RESYNC 7 mirrors 000833 1713425396.468287 write 0x00714c thru 0x00bce8 (0x4b9d bytes) 000835 1713425396.499203 trunc from 0x02c056 to 0x017118 (0x14f3d bytes) 000838 1713425396.509693 read 0x00543e thru 0x009eb4 (0x4a77 bytes) 000839 1713425396.513149 mapread 0x011589 thru 0x015c32 (0x46aa bytes) 000841 1713425396.518601 mapread 0x0050d1 thru 0x00de03 (0x8d33 bytes) 000842 1713425396.524744 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000843 1713425396.599525 write 0x01c90d thru 0x024ca3 (0x8397 bytes) 000844 1713425396.600633 read 0x01fb87 thru 0x01fffe (0x478 bytes) 000845 1713425396.601040 mapread 0x01ac4e thru 0x024ca3 (0xa056 bytes) 000846 1713425396.605709 write 0x023fb8 thru 0x02de6b (0x9eb4 bytes) 000848 1713425396.607299 mapwrite 0x021b95 thru 0x0266c8 (0x4b34 bytes) 000849 1713425396.636872 write 0x003037 thru 0x010688 (0xd652 bytes) 000850 1713425396.641753 mapwrite 0x02d164 thru 0x0394b8 (0xc355 bytes) 000851 1713425396.686082 trunc from 0x0394b9 to 0x03b003 (0x1b4b bytes) 000854 1713425396.693698 write 0x021512 thru 0x0217c6 (0x2b5 bytes) 000855 1713425396.694544 mapread 0x027fd3 thru 0x029683 (0x16b1 bytes) 000856 1713425396.695172 read 0x016bfd thru 0x02499a (0xdd9e bytes) 000857 1713425396.701811 MIRROR_EXTEND to 7 mirrors 000858 1713425396.902775 MIRROR_EXTEND to 8 mirrors 000859 1713425397.116540 mapwrite 0x004e56 thru 0x00862e (0x37d9 bytes) 000862 1713425397.180095 read 0x02ad84 thru 0x02fb96 (0x4e13 bytes) 000863 1713425397.183812 write 0x02834c thru 0x02b16f (0x2e24 bytes) 000864 1713425397.187073 trunc from 0x03b004 to 0x0357b4 (0x584f bytes) 000866 1713425397.202097 read 0x0324e2 thru 0x0324ee (0xd bytes) 000867 1713425397.206834 read 0x00f776 thru 0x01b92e (0xc1b9 bytes) 000868 1713425397.207539 trunc from 0x0357b5 to 0x022072 (0x13742 bytes) 000870 1713425397.214707 trunc from 0x022073 to 0x028a92 (0x6a20 bytes) 000877 1713425397.221241 read 0x0120d8 thru 0x0206d0 (0xe5f9 bytes) 000878 1713425397.221673 mapwrite 0x0047ad thru 0x00a7f4 (0x6048 bytes) 000879 1713425397.245687 read 0x023a5f thru 0x0251a4 (0x1746 bytes) 000882 1713425397.251975 MIRROR_EXTEND to 9 mirrors 000883 1713425397.428593 write 0x03b634 thru 0x03ffff (0x49cc bytes) 000885 1713425397.454076 mapwrite 0x012000 thru 0x01df11 (0xbf12 bytes) 000887 1713425397.493739 mapread 0x01c81a thru 0x0280a6 (0xb88d bytes) 000888 1713425397.497619 write 0x025918 thru 0x02840c (0x2af5 bytes) 000889 1713425397.498161 trunc from 0x040000 to 0x02d42d (0x12bd2 bytes) 000890 1713425397.510861 trunc from 0x02d42e to 0x0223ab (0xb082 bytes) 000892 1713425397.516094 write 0x033c01 thru 0x038133 (0x4533 bytes) 000893 1713425397.516751 read 0x0334a8 thru 0x038133 (0x4c8c bytes) 000894 1713425397.517038 read 0x033d41 thru 0x038133 (0x43f3 bytes) 000895 1713425397.518792 MIRROR_RESYNC 9 mirrors 000896 1713425397.896673 MIRROR_RESYNC 9 mirrors 000898 1713425397.953668 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000899 1713425398.033980 mapread 0x0084cd thru 0x012f15 (0xaa49 bytes) 000900 1713425398.041686 write 0x01a71c thru 0x0287be (0xe0a3 bytes) 000901 1713425398.066898 mapread 0x012fe8 thru 0x0147d5 (0x17ee bytes) 000902 1713425398.069603 read 0x011dfe thru 0x014699 (0x289c bytes) 000904 1713425398.070238 mapread 0x03463f thru 0x0373fc (0x2dbe bytes) 000906 1713425398.072948 write 0x039d0f thru 0x03f126 (0x5418 bytes) 000910 1713425398.074186 mapread 0x02b679 thru 0x039660 (0xdfe8 bytes) 000911 1713425398.076717 read 0x0181ba thru 0x01e390 (0x61d7 bytes) 000913 1713425398.077207 read 0x031118 thru 0x0383f5 (0x72de bytes) 000917 1713425398.079898 MIRROR_RESYNC 8 mirrors 000918 1713425398.304811 MIRROR_RESYNC 8 mirrors 000919 1713425398.358637 read 0x03983c thru 0x03f126 (0x58eb bytes) 000920 1713425398.360934 read 0x01c8ac thru 0x026f07 (0xa65c bytes) 000921 1713425398.363605 read 0x0067f0 thru 0x00f810 (0x9021 bytes) 000922 1713425398.363792 mapwrite 0x00225b thru 0x007024 (0x4dca bytes) 000925 1713425398.415168 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000926 1713425398.463839 mapwrite 0x01a18f thru 0x0208ff (0x6771 bytes) 000928 1713425398.482631 read 0x03d39e thru 0x03f126 (0x1d89 bytes) 000929 1713425398.484686 mapwrite 0x03d188 thru 0x03d56e (0x3e7 bytes) 000930 1713425398.497496 mapread 0x002607 thru 0x011545 (0xef3f bytes) 000932 1713425398.502398 mapread 0x029e02 thru 0x033629 (0x9828 bytes) 000935 1713425398.507747 write 0x02fce8 thru 0x031abc (0x1dd5 bytes) 000936 1713425398.508238 trunc from 0x03f127 to 0x01088d (0x2e899 bytes) 000937 1713425398.516293 trunc from 0x01088e to 0x007e0e (0x8a7f bytes) 000938 1713425398.520119 write 0x004818 thru 0x0135d6 (0xedbf bytes) 000940 1713425398.521141 mapwrite 0x035839 thru 0x036adf (0x12a7 bytes) 000941 1713425398.543661 mapwrite 0x0363ff thru 0x03bf0c (0x5b0e bytes) 000943 1713425398.563925 trunc from 0x03bf0d to 0x002a98 (0x39474 bytes) 000944 1713425398.567746 mapwrite 0x013298 thru 0x015b8d (0x28f6 bytes) 000946 1713425398.589223 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000947 1713425398.633859 write 0x020539 thru 0x028ad0 (0x8598 bytes) 000948 1713425398.634880 mapread 0x0164af thru 0x0164b0 (0x2 bytes) 000950 1713425398.638564 MIRROR_EXTEND to 7 mirrors 000951 1713425398.761416 mapread 0x00cc95 thru 0x016511 (0x987d bytes) 000952 1713425398.770419 read 0x0160a3 thru 0x01b799 (0x56f7 bytes) 000954 1713425398.772641 trunc from 0x028ad1 to 0x02b163 (0x2693 bytes) 000955 1713425398.791709 mapwrite 0x00dbda thru 0x01423c (0x6663 bytes) 000956 1713425398.812681 trunc from 0x02b164 to 0x02769e (0x3ac5 bytes) 000957 1713425398.817993 read 0x02489d thru 0x02769e (0x2e02 bytes) 000958 1713425398.820852 read 0x0225cb thru 0x0248a3 (0x22d9 bytes) 000959 1713425398.822255 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000960 1713425398.878610 mapwrite 0x0394e1 thru 0x03ffff (0x6b1f bytes) 000961 1713425398.903413 mapwrite 0x01cd2b thru 0x026a73 (0x9d49 bytes) 000965 1713425398.923032 mapread 0x01020d thru 0x019714 (0x9508 bytes) 000967 1713425398.926734 mapread 0x03fc2e thru 0x03ffff (0x3d2 bytes) 000969 1713425398.927609 mapread 0x007880 thru 0x009958 (0x20d9 bytes) 000972 1713425398.931620 mapread 0x000545 thru 0x006978 (0x6434 bytes) 000973 1713425398.934783 read 0x01eff5 thru 0x0266a7 (0x76b3 bytes) 000974 1713425398.935136 mapread 0x036ec6 thru 0x03ffff (0x913a bytes) 000978 1713425398.936870 mapwrite 0x018fa7 thru 0x01b457 (0x24b1 bytes) 000981 1713425398.954920 mapread 0x00caeb thru 0x018764 (0xbc7a bytes) 000982 1713425398.955571 trunc from 0x040000 to 0x011971 (0x2e68e bytes) 000983 1713425398.960922 write 0x012113 thru 0x01b7f5 (0x96e3 bytes) 000984 1713425398.963708 MIRROR_RESYNC 6 mirrors 000986 1713425399.189200 trunc from 0x01b7f6 to 0x00aa45 (0x10db0 bytes) 000987 1713425399.215974 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000988 1713425399.295629 write 0x0243d2 thru 0x02e60d (0xa23c bytes) 000991 1713425399.298056 mapread 0x02b4fa thru 0x02e60d (0x3114 bytes) 000992 1713425399.298588 write 0x013c41 thru 0x020251 (0xc611 bytes) 000994 1713425399.303643 write 0x00e05d thru 0x01b813 (0xd7b7 bytes) 000996 1713425399.309191 mapread 0x01d63a thru 0x026aa3 (0x946a bytes) 000997 1713425399.312652 mapwrite 0x024e0a thru 0x02b98c (0x6b83 bytes) 000998 1713425399.340260 mapread 0x02c253 thru 0x02e60d (0x23bb bytes) 001000 1713425399.343449 MIRROR_EXTEND to 6 mirrors All operations completed A-OK! PASS 70a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 03:30:02 (1713425402) Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre2 Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1904576 .. Extending file size to 6194944 .. done Extending file size to 8392224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8280352 .. Extending file size to 8582144 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8700416 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8849760 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9254432 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8724224 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9123136 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 5239 5240 5241 5243 5246 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) 192.168.201.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg114-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:31:08 (1713425468) Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre2 Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4998272 .. Extending file size to 212832 .. Extending file size to 4009984 .. Extending file size to 5261920 .. done Extending file size to 5758176 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6812864 .. Extending file size to 7983712 .. failed Extending file size to 8467328 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8691840 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8902496 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8873888 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9096256 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 8520 8521 8522 8524 8526 done 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) 192.168.201.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg114-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:32:16 (1713425536) Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0406412 s, 51.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.102472 s, 20.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:32:23 (1713425543) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0463401 s, 45.3 MB/s PASS 202 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 03:32:28 (1713425548) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaf9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaf9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xafa:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xaf4:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:32:32 (1713425552) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0440394 s, 47.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xafb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaf5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaf6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:32:36 (1713425556) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.02571 s, 81.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xafc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xafd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaf7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xafc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xafd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaf7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaf8:0x0] } PASS 204b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:32:41 (1713425561) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0427164 s, 49.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xafe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaf9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xafe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaf9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafa:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:33:07 (1713425587) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0240389 s, 43.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaff:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaff:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb00:0x0] } PASS 204d (45s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:33:54 (1713425634) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0439861 s, 47.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb01:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xafe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb01:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:34:44 (1713425684) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. Creating new pool oleg114-server: Pool lustre.test_204f created Adding targets to pool oleg114-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg114-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0227294 s, 92.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb02:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb02:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg114-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg114-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg114-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 03:35:48 (1713425748) PASS 205 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:35:52 (1713425752) oleg114-server: Pool lustre.test_206 created Adding targets to pool oleg114-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg114-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg114-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg114-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg114-server: Pool lustre.test_206 destroyed PASS 206 (15s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:36:09 (1713425769) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296995 s, 35.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148511 s, 70.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x8c6c443a Mirror 3: 0x8c6c443a PASS 207 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:36:15 (1713425775) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:36:18 (1713425778) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:36:21 (1713425781) Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0307822 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1030000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1df0000 Read this (4 bytes): BBBB 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:36:39 (1713425799) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00229878 s, 3.6 MB/s the very first read read with warm cache PASS 209b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 03:36:42 (1713425802) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115642 s, 90.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5e3 lmm_fid: [0x200000404:0x5e3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5e3 lmm_fid: [0x200000404:0x5e3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb08:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835427 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-18 03:36:43.000000000 -0400 Modify: 2024-04-18 03:36:43.000000000 -0400 Change: 2024-04-18 03:36:43.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:36:46 (1713425806) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 812 sec ============ 03:36:47 (1713425807) === sanity-flr: start cleanup 03:36:48 (1713425808) === === sanity-flr: finish cleanup 03:36:48 (1713425808) ===