-----============= acceptance-small: sanity-flr ============----- Wed Apr 17 10:29:39 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:29:44 (1713364184) === oleg450-client.virtnet: executing check_config_client /mnt/lustre oleg450-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg450-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5822800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5822800.idle_timeout=debug disable quota as required oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:29:52 (1713364192) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6916] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 10:29:53 (1713364193) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:29:57 (1713364197) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:30:00 (1713364200) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 10:30:02 (1713364202) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:30:08 (1713364208) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:30:10 (1713364210) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 10:30:12 (1713364212) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5822800.stats=clear osc.lustre-OST0001-osc-ffff8800b5822800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:30:16 (1713364216) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:30:19 (1713364219) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:30:22 (1713364222) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 10:30:24 (1713364224) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 10:30:28 (1713364228) PASS 2 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:30:31 (1713364231) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:30:33 (1713364233) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:30:36 (1713364236) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 10:30:40 (1713364240) PASS 7 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:30:43 (1713364243) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0501917 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0397103 s, 26.4 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:30:46 (1713364246) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.100489 s, 198 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:30:50 (1713364250) Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5822800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5822800.ost_server_uuid in DISCONN state after 4 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5822800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5822800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:31:13 (1713364273) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0798074 s, 65.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5822800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5822800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:31:27 (1713364287) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5822800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5822800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5822800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5822800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:31:42 (1713364302) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.800454 s, 65.5 MB/s -rw-r--r-- 1 root root 52428800 Apr 17 10:31 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.196s user 0m0.001s sys 0m0.119s Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5822800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5822800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.185s user 0m0.000s sys 0m0.112s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5822800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5822800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.189s user 0m0.000s sys 0m0.107s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5822800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5822800.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:32:03 (1713364323) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:32:06 (1713364326) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:32:08 (1713364328) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 10:32:10 (1713364330) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 10:32:13 (1713364333) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.115725 s, 109 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:32:18 (1713364338) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00683959 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00752776 s, 139 MB/s PASS 36b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:32:21 (1713364341) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:32:25 (1713364345) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000704103 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000475179 s, 2.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000517107 s, 2.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00060607 s, 1.7 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 10:32:29 (1713364349) create test files with size 15754252 .. a6b0d7604b51cd53007465dc3b197aef - 3cc257719e8e3662fb4e5eed09763ac0 - 8f3ee01fb5639ba5f60cbfad76b3582e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 10:32:35 (1713364355) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:32:39 (1713364359) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:32:42 (1713364362) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192714 s, 54.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00799128 s, 131 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 10:32:45 (1713364365) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0240919 s, 87.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0840509 s, 49.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0374136 s, 56.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 10:32:49 (1713364369) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 10:32:52 (1713364372) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:32:54 (1713364374) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00767476 s, 137 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 10:32:57 (1713364377) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:32:59 (1713364379) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:33:03 (1713364383) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.05751 s, 182 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:33:07 (1713364387) llite.lustre-ffff88012a47e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0575307 s, 182 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800b5824000.xattr_cache=1 PASS 44e (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:33:11 (1713364391) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 10:33:19 (1713364399) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 10:33:25 (1713364405) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 10:33:27 (1713364407) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0556358 s, 56.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0474741 s, 66.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: b51fd8eb58dc040586128bc9277b0a3b - mirror 1 checksum: 92f2c04e602a6d3544e3c10bf0276d2e - mirror 2 checksum: b51fd8eb58dc040586128bc9277b0a3b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 17 10:33:29 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 17 10:33:29 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b51fd8eb58dc040586128bc9277b0a3b - mirror 1 checksum: 92f2c04e602a6d3544e3c10bf0276d2e - mirror 2 checksum: 92f2c04e602a6d3544e3c10bf0276d2e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 17 10:33:29 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 17 10:33:29 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (3s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:33:32 (1713364412) Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server 10:33:33 (1713364413) shut down Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 10:33:46 (1713364426) targets are mounted 10:33:46 (1713364426) facet_failover done oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00736003 s, 142 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:33:52 (1713364432) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 10:33:54 (1713364434) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:33:56 (1713364436) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00196961 s, 8.3 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 10:34:00 (1713364440) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:34:02 (1713364442) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:34:04 (1713364444) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:34:10 (1713364450) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:34:30 (1713364470) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:34:48 (1713364488) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:35:07 (1713364507) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 10:35:10 (1713364510) mirror create pid 26645 mirror split pid 26647 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6537: 26647 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6537: 26647 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4710: 26645 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4710: 26647 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 10:36:14 (1713364574) Seed set to 4574 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x124a7 000001 1713364574.565806 trunc from 00000000 to 0x0124a6 (0x124a7 bytes) truncating to largest ever: 0x1d2af 000002 1713364574.570716 trunc from 0x0124a7 to 0x01d2ae (0xae08 bytes) 000003 1713364574.575856 read 0x005eb8 thru 0x0131ec (0xd335 bytes) 000004 1713364574.579381 mapread 0x014fc3 thru 0x0172bc (0x22fa bytes) 000006 1713364574.585369 MIRROR_EXTEND to 2 mirrors 000009 1713364574.716339 mapread 0x01bef6 thru 0x01d2ae (0x13b9 bytes) truncating to largest ever: 0x39de4 000010 1713364574.720859 trunc from 0x01d2af to 0x039de3 (0x1cb35 bytes) 000013 1713364574.741095 mapwrite 0x02ed76 thru 0x0309de (0x1c69 bytes) 000014 1713364574.758078 write 0x02cd06 thru 0x0342a9 (0x75a4 bytes) 000015 1713364574.758909 trunc from 0x039de4 to 0x00d92c (0x2c4b7 bytes) 000019 1713364574.766583 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000020 1713364574.802161 write 0x029ab1 thru 0x0315d1 (0x7b21 bytes) 000022 1713364574.803573 write 0x02f325 thru 0x03b120 (0xbdfc bytes) 000023 1713364574.804480 trunc from 0x03b121 to 0x014138 (0x26fe8 bytes) 000024 1713364574.809811 mapwrite 0x0355a8 thru 0x03ffff (0xaa58 bytes) 000025 1713364574.835417 mapread 0x0312e1 thru 0x03a2e7 (0x9007 bytes) 000027 1713364574.836501 mapread 0x031651 thru 0x03a973 (0x9323 bytes) 000029 1713364574.837302 mapread 0x01b772 thru 0x01d013 (0x18a2 bytes) 000030 1713364574.840064 mapwrite 0x023149 thru 0x02d9b8 (0xa870 bytes) 000033 1713364574.858095 read 0x00e82a thru 0x013943 (0x511a bytes) 000035 1713364574.860987 write 0x021682 thru 0x0290cb (0x7a4a bytes) 000036 1713364574.861756 write 0x00aa7e thru 0x01244f (0x79d2 bytes) 000037 1713364574.864670 mapread 0x018f52 thru 0x01d5d0 (0x467f bytes) 000039 1713364574.865617 trunc from 0x040000 to 0x028477 (0x17b88 bytes) 000040 1713364574.869852 mapwrite 0x00de81 thru 0x014575 (0x66f5 bytes) 000041 1713364574.887393 trunc from 0x028478 to 0x0240d6 (0x43a1 bytes) 000042 1713364574.891524 read 0x00b513 thru 0x015b03 (0xa5f1 bytes) 000044 1713364574.892281 read 0x00a6c1 thru 0x0151bb (0xaafb bytes) 000047 1713364574.893522 read 0x00460a thru 0x00d193 (0x8b8a bytes) 000048 1713364574.893794 mapread 0x01e24b thru 0x0240d6 (0x5e8c bytes) 000050 1713364574.894701 mapread 0x01fe05 thru 0x0240d6 (0x42d2 bytes) 000053 1713364574.895719 mapwrite 0x018a4e thru 0x025c54 (0xd207 bytes) 000060 1713364574.928555 mapwrite 0x00f574 thru 0x0155ff (0x608c bytes) 000062 1713364574.948773 read 0x0165c2 thru 0x0173f3 (0xe32 bytes) 000067 1713364574.950638 mapwrite 0x012e0c thru 0x02165e (0xe853 bytes) 000069 1713364574.967549 read 0x008fc3 thru 0x009640 (0x67e bytes) 000070 1713364574.967793 mapwrite 0x03e016 thru 0x03e219 (0x204 bytes) 000072 1713364574.990800 mapwrite 0x01f127 thru 0x025717 (0x65f1 bytes) 000074 1713364575.008087 mapwrite 0x01b7ea thru 0x021113 (0x592a bytes) 000075 1713364575.027472 write 0x019768 thru 0x021e2a (0x86c3 bytes) 000077 1713364575.028607 mapread 0x01707b thru 0x0194db (0x2461 bytes) 000078 1713364575.029000 write 0x02aa5d thru 0x035caa (0xb24e bytes) 000081 1713364575.030547 write 0x02a0c9 thru 0x039b0d (0xfa45 bytes) 000084 1713364575.031878 mapread 0x0227ea thru 0x023448 (0xc5f bytes) 000087 1713364575.032972 write 0x032d75 thru 0x0368a4 (0x3b30 bytes) 000089 1713364575.033842 write 0x02db5b thru 0x03b5fe (0xdaa4 bytes) 000090 1713364575.034735 mapread 0x0042f9 thru 0x011b62 (0xd86a bytes) 000094 1713364575.036134 mapwrite 0x03fc01 thru 0x03ffff (0x3ff bytes) 000095 1713364575.061412 mapwrite 0x02b0f8 thru 0x02b43f (0x348 bytes) 000097 1713364575.076858 MIRROR_EXTEND to 2 mirrors 000099 1713364575.206483 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000100 1713364575.238866 mapread 0x011794 thru 0x017911 (0x617e bytes) 000101 1713364575.245052 write 0x012e5e thru 0x019e36 (0x6fd9 bytes) 000102 1713364575.252248 write 0x03c0e6 thru 0x03ffff (0x3f1a bytes) 000103 1713364575.255115 mapwrite 0x019f99 thru 0x01d777 (0x37df bytes) 000104 1713364575.278629 mapwrite 0x033ead thru 0x0361a5 (0x22f9 bytes) 000105 1713364575.295725 mapread 0x02337a thru 0x023ca3 (0x92a bytes) 000107 1713364575.296600 mapwrite 0x03c2b0 thru 0x03ffff (0x3d50 bytes) 000108 1713364575.314879 write 0x015342 thru 0x0169dc (0x169b bytes) 000111 1713364575.316585 mapread 0x001675 thru 0x00abef (0x957b bytes) 000112 1713364575.319547 mapread 0x012fbf thru 0x01f26f (0xc2b1 bytes) 000113 1713364575.321068 MIRROR_EXTEND to 2 mirrors 000114 1713364575.454672 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000115 1713364575.483093 read 0x03d3a1 thru 0x03fa44 (0x26a4 bytes) 000116 1713364575.485038 trunc from 0x040000 to 0x02f6b1 (0x1094e bytes) 000120 1713364575.493046 read 0x0171c9 thru 0x018d58 (0x1b90 bytes) 000121 1713364575.496324 read 0x0132ed thru 0x0199e2 (0x66f6 bytes) 000123 1713364575.496988 mapwrite 0x03e1e8 thru 0x03ffff (0x1e18 bytes) 000124 1713364575.517286 read 0x02c3dc thru 0x03342c (0x7051 bytes) 000126 1713364575.517987 write 0x018ee4 thru 0x020d87 (0x7ea4 bytes) 000129 1713364575.519115 write 0x01e9a7 thru 0x022794 (0x3dee bytes) 000131 1713364575.520316 mapread 0x003e5a thru 0x00439e (0x545 bytes) 000132 1713364575.520871 mapread 0x005476 thru 0x006352 (0xedd bytes) 000133 1713364575.522958 MIRROR_EXTEND to 2 mirrors 000134 1713364575.662137 trunc from 0x040000 to 0x008815 (0x377ea bytes) 000135 1713364575.684473 trunc from 0x008816 to 0x036d58 (0x2e543 bytes) 000137 1713364575.692530 mapwrite 0x000543 thru 0x000ba6 (0x664 bytes) 000138 1713364575.712197 MIRROR_RESYNC 2 mirrors 000139 1713364575.815612 mapread 0x02a64d thru 0x02f1c1 (0x4b75 bytes) 000142 1713364575.820699 mapwrite 0x02032a thru 0x02e242 (0xdf19 bytes) 000144 1713364575.869730 trunc from 0x036d59 to 0x017653 (0x1f705 bytes) 000145 1713364575.878160 MIRROR_EXTEND to 3 mirrors 000147 1713364575.993795 mapwrite 0x00fda2 thru 0x016eed (0x714c bytes) 000148 1713364576.052926 write 0x022654 thru 0x023acf (0x147c bytes) 000149 1713364576.056360 MIRROR_RESYNC 3 mirrors 000150 1713364576.202876 read 0x017aea thru 0x023acf (0xbfe6 bytes) 000151 1713364576.207128 MIRROR_RESYNC 3 mirrors 000152 1713364576.246645 mapread 0x0067f5 thru 0x00ecfe (0x850a bytes) 000153 1713364576.250792 mapwrite 0x023651 thru 0x02960d (0x5fbd bytes) 000154 1713364576.282929 write 0x01ae00 thru 0x01c150 (0x1351 bytes) 000157 1713364576.286128 write 0x03b57e thru 0x03ffff (0x4a82 bytes) 000158 1713364576.286649 read 0x00dc33 thru 0x01989d (0xbc6b bytes) 000161 1713364576.289333 mapwrite 0x021471 thru 0x02c1ee (0xad7e bytes) 000162 1713364576.314610 mapwrite 0x02b195 thru 0x035d70 (0xabdc bytes) 000164 1713364576.342190 mapwrite 0x030012 thru 0x0319cb (0x19ba bytes) 000167 1713364576.359036 write 0x00808a thru 0x008138 (0xaf bytes) 000168 1713364576.362148 read 0x01288b thru 0x0136b2 (0xe28 bytes) 000169 1713364576.362576 mapread 0x012ae3 thru 0x01c7d0 (0x9cee bytes) 000170 1713364576.364710 MIRROR_RESYNC 3 mirrors 000171 1713364576.488118 MIRROR_EXTEND to 4 mirrors 000172 1713364576.618261 trunc from 0x040000 to 0x0130fa (0x2cf05 bytes) 000174 1713364576.646235 read 0x00556b thru 0x00e5fa (0x9090 bytes) 000175 1713364576.649420 write 0x007724 thru 0x012658 (0xaf35 bytes) 000178 1713364576.653654 MIRROR_EXTEND to 5 mirrors 000180 1713364576.783770 mapwrite 0x03f0ca thru 0x03ffff (0xf36 bytes) 000181 1713364576.820810 MIRROR_RESYNC 5 mirrors 000182 1713364576.987648 trunc from 0x040000 to 0x01662c (0x299d3 bytes) 000183 1713364577.009126 trunc from 0x01662d to 0x0020b1 (0x1457b bytes) 000185 1713364577.015347 write 0x0379ec thru 0x03ffff (0x8614 bytes) 000186 1713364577.017876 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000188 1713364577.073185 mapwrite 0x00defe thru 0x01b367 (0xd46a bytes) 000189 1713364577.098397 read 0x00d5d9 thru 0x018b67 (0xb58f bytes) 000190 1713364577.098851 read 0x0026df thru 0x002deb (0x70d bytes) 000191 1713364577.104416 trunc from 0x040000 to 0x00557d (0x3aa82 bytes) 000192 1713364577.113067 read 0x003169 thru 0x00557d (0x2415 bytes) 000193 1713364577.113554 read 0x001f08 thru 0x00557d (0x3676 bytes) 000195 1713364577.114237 read 0x003e22 thru 0x00557d (0x175c bytes) 000196 1713364577.114659 write 0x02f0cb thru 0x032012 (0x2f48 bytes) 000197 1713364577.115275 trunc from 0x032013 to 0x015e2c (0x1c1e6 bytes) 000202 1713364577.120412 write 0x03628d thru 0x03a374 (0x40e8 bytes) 000203 1713364577.121021 read 0x00e47a thru 0x012127 (0x3cae bytes) 000204 1713364577.124841 mapread 0x00ce67 thru 0x00f2bd (0x2457 bytes) 000205 1713364577.125432 trunc from 0x03a375 to 0x00dde1 (0x2c593 bytes) 000206 1713364577.129982 read 0x00a191 thru 0x00dde1 (0x3c51 bytes) 000207 1713364577.130353 trunc from 0x00dde2 to 0x0271fe (0x1941d bytes) 000208 1713364577.135054 mapwrite 0x022f61 thru 0x0295d1 (0x6671 bytes) 000209 1713364577.161893 mapread 0x011a29 thru 0x01fe51 (0xe429 bytes) 000211 1713364577.162832 trunc from 0x0295d2 to 0x023bfa (0x59d7 bytes) 000215 1713364577.167477 read 0x02008c thru 0x023bfa (0x3b6f bytes) 000218 1713364577.168155 trunc from 0x023bfb to 0x01f864 (0x4396 bytes) 000219 1713364577.171990 trunc from 0x01f865 to 0x0045be (0x1b2a6 bytes) 000220 1713364577.175301 write 0x03e53d thru 0x03ffff (0x1ac3 bytes) 000221 1713364577.175752 read 0x015b55 thru 0x01b50e (0x59ba bytes) 000222 1713364577.178545 trunc from 0x040000 to 0x00705a (0x38fa5 bytes) 000223 1713364577.182136 trunc from 0x00705b to 0x004f65 (0x20f5 bytes) 000224 1713364577.185481 write 0x01f6e8 thru 0x0263bb (0x6cd4 bytes) 000225 1713364577.186292 read 0x00044d thru 0x005a9d (0x5651 bytes) 000226 1713364577.188601 read 0x009275 thru 0x0175c3 (0xe34f bytes) 000227 1713364577.188957 mapwrite 0x020383 thru 0x0205e2 (0x260 bytes) 000228 1713364577.204707 read 0x014860 thru 0x022a8a (0xe22b bytes) 000229 1713364577.205033 trunc from 0x0263bc to 0x023435 (0x2f86 bytes) 000230 1713364577.208993 trunc from 0x023436 to 0x001725 (0x21d10 bytes) 000231 1713364577.212626 mapread 0x00006d thru 0x001725 (0x16b9 bytes) 000233 1713364577.213513 write 0x002673 thru 0x0068cd (0x425b bytes) 000234 1713364577.214077 trunc from 0x0068ce to 0x01cf40 (0x16673 bytes) 000235 1713364577.218977 MIRROR_EXTEND to 5 mirrors 000236 1713364577.348705 mapread 0x0044d1 thru 0x013012 (0xeb42 bytes) 000238 1713364577.363745 MIRROR_EXTEND to 6 mirrors 000240 1713364577.479232 trunc from 0x01cf41 to 0x0294e0 (0xc5a0 bytes) 000241 1713364577.503499 mapread 0x00de2a thru 0x010b81 (0x2d58 bytes) 000242 1713364577.508938 write 0x004f90 thru 0x007df1 (0x2e62 bytes) 000244 1713364577.511720 mapread 0x00a12a thru 0x014e68 (0xad3f bytes) 000245 1713364577.512345 write 0x03563c thru 0x03d903 (0x82c8 bytes) 000248 1713364577.513862 trunc from 0x03d904 to 0x011a4c (0x2beb7 bytes) 000250 1713364577.521873 MIRROR_EXTEND to 7 mirrors 000251 1713364577.647358 mapread 0x002531 thru 0x005205 (0x2cd5 bytes) 000252 1713364577.654075 write 0x03af4d thru 0x03c3f0 (0x14a4 bytes) 000253 1713364577.671604 mapwrite 0x036d81 thru 0x0399c3 (0x2c43 bytes) 000254 1713364577.694898 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000257 1713364577.752789 MIRROR_RESYNC 6 mirrors 000258 1713364578.030704 mapread 0x03b9cc thru 0x03c3f0 (0xa25 bytes) 000260 1713364578.034242 trunc from 0x03c3f1 to 0x02f356 (0xd09a bytes) 000261 1713364578.058618 trunc from 0x02f357 to 0x03932a (0x9fd4 bytes) 000263 1713364578.064978 mapread 0x00e784 thru 0x016765 (0x7fe2 bytes) 000265 1713364578.071715 MIRROR_EXTEND to 7 mirrors 000266 1713364578.228332 trunc from 0x03932b to 0x011560 (0x27dca bytes) 000268 1713364578.252544 mapread 0x004680 thru 0x0061b5 (0x1b36 bytes) 000269 1713364578.258066 MIRROR_RESYNC 7 mirrors 000270 1713364578.508862 mapread 0x007446 thru 0x00d227 (0x5de2 bytes) 000274 1713364578.515562 mapread 0x002974 thru 0x00d190 (0xa81d bytes) 000276 1713364578.517881 mapread 0x00e2a3 thru 0x011560 (0x32be bytes) 000278 1713364578.518402 mapwrite 0x020f3c thru 0x028dfd (0x7ec2 bytes) 000280 1713364578.570315 mapwrite 0x01f696 thru 0x027e54 (0x87bf bytes) 000281 1713364578.587410 read 0x019681 thru 0x02399d (0xa31d bytes) 000282 1713364578.590762 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000283 1713364578.648989 mapwrite 0x007149 thru 0x012c80 (0xbb38 bytes) 000288 1713364578.678836 mapwrite 0x00b9ff thru 0x0153db (0x99dd bytes) 000289 1713364578.703460 write 0x02ea43 thru 0x03815d (0x971b bytes) 000292 1713364578.705006 write 0x019922 thru 0x023ba8 (0xa287 bytes) 000296 1713364578.709767 write 0x004133 thru 0x00427f (0x14d bytes) 000297 1713364578.710359 mapread 0x021ff3 thru 0x027e09 (0x5e17 bytes) 000298 1713364578.717057 MIRROR_RESYNC 6 mirrors 000300 1713364578.998306 trunc from 0x03815e to 0x003b29 (0x34634 bytes) 000301 1713364579.027597 mapread 0x0025c8 thru 0x003b29 (0x1562 bytes) 000304 1713364579.033815 write 0x001f53 thru 0x011652 (0xf700 bytes) 000305 1713364579.036939 read 0x0022fc thru 0x00317b (0xe80 bytes) 000308 1713364579.040773 MIRROR_RESYNC 6 mirrors 000310 1713364579.337520 read 0x00655b thru 0x009087 (0x2b2d bytes) 000313 1713364579.339555 read 0x006802 thru 0x011652 (0xae51 bytes) 000315 1713364579.342957 read 0x0003ca thru 0x00bb6f (0xb7a6 bytes) 000317 1713364579.343341 read 0x000e28 thru 0x00b88f (0xaa68 bytes) 000318 1713364579.343595 trunc from 0x011653 to 0x00fe89 (0x17c9 bytes) 000319 1713364579.376925 mapread 0x008509 thru 0x00fe89 (0x7981 bytes) 000320 1713364579.379864 read 0x00d622 thru 0x00fe89 (0x2868 bytes) 000321 1713364579.381390 MIRROR_RESYNC 6 mirrors 000322 1713364579.621498 mapread 0x00259a thru 0x00fe89 (0xd8f0 bytes) 000324 1713364579.631084 read 0x00691f thru 0x00aed9 (0x45bb bytes) 000325 1713364579.631388 read 0x00a55c thru 0x00fe89 (0x592e bytes) 000326 1713364579.631613 read 0x00ae62 thru 0x00fe89 (0x5028 bytes) 000327 1713364579.631803 mapwrite 0x0215f9 thru 0x02c78b (0xb193 bytes) 000330 1713364579.703976 mapwrite 0x004409 thru 0x004de4 (0x9dc bytes) 000335 1713364579.725223 write 0x032c9d thru 0x038ac7 (0x5e2b bytes) 000336 1713364579.726056 write 0x030304 thru 0x033fe6 (0x3ce3 bytes) 000338 1713364579.730834 mapwrite 0x02ef79 thru 0x03dcd6 (0xed5e bytes) 000342 1713364579.767886 mapwrite 0x032ec0 thru 0x03ffff (0xd140 bytes) 000343 1713364579.797376 read 0x01d5d8 thru 0x02b0ff (0xdb28 bytes) 000346 1713364579.798401 read 0x031aca thru 0x03ded2 (0xc409 bytes) 000348 1713364579.799040 read 0x0339f5 thru 0x03ffff (0xc60b bytes) 000349 1713364579.799416 write 0x03e718 thru 0x03e92e (0x217 bytes) 000350 1713364579.799954 mapwrite 0x036ee6 thru 0x03e29b (0x73b6 bytes) 000352 1713364579.823166 write 0x02f993 thru 0x030df6 (0x1464 bytes) 000353 1713364579.825135 MIRROR_RESYNC 6 mirrors 000354 1713364580.046140 read 0x02e7c7 thru 0x037f7f (0x97b9 bytes) 000356 1713364580.048637 read 0x02db3c thru 0x033a18 (0x5edd bytes) 000360 1713364580.052756 read 0x0166e9 thru 0x022762 (0xc07a bytes) 000361 1713364580.053628 read 0x00c5c4 thru 0x015689 (0x90c6 bytes) 000362 1713364580.053978 mapwrite 0x03817f thru 0x03839b (0x21d bytes) 000363 1713364580.116864 MIRROR_RESYNC 6 mirrors 000366 1713364580.382999 mapwrite 0x02ab99 thru 0x0326a6 (0x7b0e bytes) 000368 1713364580.449524 MIRROR_RESYNC 6 mirrors 000369 1713364580.696187 MIRROR_EXTEND to 7 mirrors 000370 1713364580.938267 trunc from 0x040000 to 0x00ed6e (0x31291 bytes) 000371 1713364580.966687 mapwrite 0x036899 thru 0x038062 (0x17ca bytes) 000372 1713364581.010001 trunc from 0x038063 to 0x00d590 (0x2aad2 bytes) 000373 1713364581.015525 write 0x023f21 thru 0x0338b4 (0xf994 bytes) 000374 1713364581.016690 mapwrite 0x0193bd thru 0x01d632 (0x4276 bytes) 000377 1713364581.041967 read 0x0330d1 thru 0x0338b4 (0x7e4 bytes) 000378 1713364581.042345 trunc from 0x0338b5 to 0x03019e (0x3716 bytes) 000379 1713364581.047521 mapwrite 0x008a62 thru 0x00c2e1 (0x3880 bytes) 000382 1713364581.072736 read 0x00fbf3 thru 0x012d8f (0x319d bytes) 000384 1713364581.076485 write 0x00183c thru 0x00cc9e (0xb463 bytes) 000387 1713364581.077971 write 0x037efa thru 0x03ffff (0x8106 bytes) 000391 1713364581.079670 trunc from 0x040000 to 0x02431e (0x1bce1 bytes) 000395 1713364581.086228 mapwrite 0x0037f5 thru 0x00c88b (0x9097 bytes) 000396 1713364581.116970 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000397 1713364581.176506 mapread 0x0213b9 thru 0x022307 (0xf4f bytes) 000402 1713364581.181334 read 0x0057e5 thru 0x0120c1 (0xc8dd bytes) 000405 1713364581.184678 mapread 0x01a1b8 thru 0x021662 (0x74ab bytes) 000406 1713364581.187382 read 0x0007fb thru 0x0039d2 (0x31d8 bytes) 000407 1713364581.190254 trunc from 0x02431f to 0x02e563 (0xa245 bytes) 000408 1713364581.196217 mapwrite 0x01c6f9 thru 0x02ab06 (0xe40e bytes) 000409 1713364581.230002 write 0x015be3 thru 0x021d24 (0xc142 bytes) 000410 1713364581.232706 MIRROR_RESYNC 6 mirrors 000411 1713364581.543461 mapread 0x01f26b thru 0x023f31 (0x4cc7 bytes) 000412 1713364581.549685 trunc from 0x02e564 to 0x033c51 (0x56ee bytes) 000413 1713364581.577119 mapwrite 0x01a4d8 thru 0x02758c (0xd0b5 bytes) 000415 1713364581.622310 trunc from 0x033c52 to 0x023d37 (0xff1a bytes) 000417 1713364581.636966 MIRROR_RESYNC 6 mirrors 000419 1713364581.936866 write 0x036b1d thru 0x03b3d2 (0x48b6 bytes) 000420 1713364581.960055 MIRROR_EXTEND to 7 mirrors 000421 1713364582.164469 write 0x0330e2 thru 0x03ffff (0xcf1e bytes) 000422 1713364582.191965 trunc from 0x040000 to 0x008506 (0x37af9 bytes) 000423 1713364582.199820 read 0x004037 thru 0x008506 (0x44d0 bytes) 000424 1713364582.203332 mapread 0x005e5d thru 0x008506 (0x26aa bytes) 000426 1713364582.204305 mapread 0x005f20 thru 0x008506 (0x25e7 bytes) 000427 1713364582.204853 mapread 0x0030ae thru 0x008506 (0x5459 bytes) 000432 1713364582.208303 trunc from 0x008507 to 0x025b86 (0x1d680 bytes) 000433 1713364582.216071 write 0x0019e2 thru 0x004ae7 (0x3106 bytes) 000435 1713364582.220698 read 0x0054c0 thru 0x00f9cb (0xa50c bytes) truncating to largest ever: 0x3c970 000437 1713364582.221436 trunc from 0x025b87 to 0x03c96f (0x16de9 bytes) 000439 1713364582.229817 MIRROR_EXTEND to 8 mirrors 000440 1713364582.425770 MIRROR_EXTEND to 9 mirrors 000441 1713364582.587588 mapwrite 0x018399 thru 0x0243ae (0xc016 bytes) 000442 1713364582.668989 MIRROR_EXTEND to 10 mirrors 000447 1713364582.821986 mapread 0x006d4b thru 0x011967 (0xac1d bytes) 000450 1713364582.835009 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000451 1713364582.896325 mapwrite 0x00d670 thru 0x01cecc (0xf85d bytes) 000452 1713364582.942241 trunc from 0x03c970 to 0x00fa7f (0x2cef0 bytes) 000454 1713364582.952819 write 0x01eabe thru 0x02bd36 (0xd279 bytes) 000455 1713364582.953851 write 0x0042ce thru 0x013786 (0xf4b9 bytes) 000456 1713364582.959079 read 0x01efd1 thru 0x02bd36 (0xcd66 bytes) 000459 1713364582.960241 read 0x00e456 thru 0x00f7c8 (0x1373 bytes) 000462 1713364582.961252 trunc from 0x02bd37 to 0x01b096 (0x10ca0 bytes) 000463 1713364582.966747 read 0x006049 thru 0x0132db (0xd293 bytes) 000464 1713364582.967183 mapwrite 0x033e15 thru 0x03856b (0x4757 bytes) 000465 1713364582.998254 mapwrite 0x03a02d thru 0x03c1a9 (0x217d bytes) 000466 1713364583.027473 write 0x003bbb thru 0x012b3a (0xef80 bytes) 000467 1713364583.028434 read 0x001c31 thru 0x00d146 (0xb516 bytes) 000470 1713364583.029419 read 0x037da2 thru 0x03c1a9 (0x4408 bytes) 000474 1713364583.030603 trunc from 0x03c1aa to 0x018c38 (0x23571 bytes) 000475 1713364583.036516 read 0x0132d0 thru 0x018c38 (0x5969 bytes) 000476 1713364583.036977 read 0x01029c thru 0x018c38 (0x899d bytes) 000477 1713364583.037272 trunc from 0x018c39 to 0x007cf7 (0x10f41 bytes) 000481 1713364583.046265 MIRROR_RESYNC 9 mirrors 000483 1713364583.439895 write 0x019742 thru 0x02087c (0x713b bytes) 000484 1713364583.462884 trunc from 0x02087d to 0x010e5f (0xfa1d bytes) 000485 1713364583.468698 write 0x00159d thru 0x002890 (0x12f4 bytes) 000486 1713364583.472252 mapread 0x001653 thru 0x005d9c (0x474a bytes) 000487 1713364583.477114 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000488 1713364583.532853 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000489 1713364583.587245 MIRROR_RESYNC 7 mirrors 000490 1713364583.854028 MIRROR_RESYNC 7 mirrors 000492 1713364583.913610 MIRROR_RESYNC 7 mirrors 000495 1713364583.965106 mapread 0x010503 thru 0x010e5f (0x95d bytes) 000496 1713364583.965534 write 0x0237ce thru 0x0257c6 (0x1ff9 bytes) 000498 1713364583.984369 mapread 0x003246 thru 0x00691c (0x36d7 bytes) 000500 1713364583.992011 MIRROR_RESYNC 7 mirrors 000502 1713364584.315514 trunc from 0x0257c7 to 0x03447e (0xecb8 bytes) 000503 1713364584.339003 MIRROR_RESYNC 7 mirrors 000505 1713364584.634678 trunc from 0x03447f to 0x019878 (0x1ac06 bytes) 000507 1713364584.656892 write 0x021e11 thru 0x023087 (0x1277 bytes) 000508 1713364584.657568 mapwrite 0x03b655 thru 0x03ffff (0x49ab bytes) 000510 1713364584.705891 write 0x004650 thru 0x00b78e (0x713f bytes) 000511 1713364584.711243 read 0x01e912 thru 0x02a02a (0xb719 bytes) 000514 1713364584.717352 write 0x023979 thru 0x023aba (0x142 bytes) 000516 1713364584.717910 mapwrite 0x00d16f thru 0x01377c (0x660e bytes) 000517 1713364584.741870 mapwrite 0x0174d4 thru 0x02592d (0xe45a bytes) 000518 1713364584.760866 write 0x01cd76 thru 0x02a426 (0xd6b1 bytes) 000519 1713364584.761753 write 0x0248e9 thru 0x02757d (0x2c95 bytes) 000520 1713364584.762196 write 0x009e34 thru 0x00af91 (0x115e bytes) 000523 1713364584.763145 trunc from 0x040000 to 0x0320e6 (0xdf19 bytes) 000529 1713364584.769619 mapread 0x00d0fd thru 0x019544 (0xc448 bytes) 000533 1713364584.772668 MIRROR_EXTEND to 8 mirrors 000534 1713364584.940105 write 0x03dd28 thru 0x03ffff (0x22d8 bytes) 000536 1713364584.962470 MIRROR_RESYNC 8 mirrors 000537 1713364585.265463 write 0x02789b thru 0x030d9c (0x9502 bytes) 000538 1713364585.290887 read 0x02f077 thru 0x03d29b (0xe225 bytes) 000539 1713364585.293545 write 0x02b17c thru 0x0366ee (0xb573 bytes) 000540 1713364585.294253 read 0x006e16 thru 0x00c8f5 (0x5ae0 bytes) 000541 1713364585.297691 mapread 0x0117b7 thru 0x0158d8 (0x4122 bytes) 000542 1713364585.298288 read 0x0367d2 thru 0x03bd62 (0x5591 bytes) 000545 1713364585.299288 write 0x01407e thru 0x015511 (0x1494 bytes) 000546 1713364585.299876 write 0x0037d3 thru 0x00d016 (0x9844 bytes) 000547 1713364585.300411 read 0x027a16 thru 0x035812 (0xddfd bytes) 000549 1713364585.301036 write 0x03d06a thru 0x03ffff (0x2f96 bytes) 000550 1713364585.301563 read 0x00e21b thru 0x010f7f (0x2d65 bytes) 000551 1713364585.301875 trunc from 0x040000 to 0x01996c (0x26693 bytes) 000552 1713364585.310523 MIRROR_EXTEND to 9 mirrors 000553 1713364585.468019 write 0x011148 thru 0x0201de (0xf097 bytes) 000554 1713364585.492393 mapread 0x01cf3f thru 0x0201de (0x32a0 bytes) 000555 1713364585.494607 MIRROR_EXTEND to 10 mirrors 000557 1713364585.646993 mapwrite 0x0002e2 thru 0x006dda (0x6af9 bytes) 000559 1713364585.712605 mapwrite 0x03145c thru 0x03b007 (0x9bac bytes) 000561 1713364585.753493 mapwrite 0x02573c thru 0x031bf0 (0xc4b5 bytes) 000562 1713364585.780945 trunc from 0x03b008 to 0x03bf5d (0xf56 bytes) 000564 1713364585.786571 read 0x01ea07 thru 0x02bd07 (0xd301 bytes) 000565 1713364585.789552 MIRROR_EXTEND to 11 mirrors 000568 1713364585.927125 MIRROR_RESYNC 11 mirrors 000569 1713364586.398266 mapread 0x01b26f thru 0x022173 (0x6f05 bytes) 000570 1713364586.404544 write 0x0098b8 thru 0x0164c0 (0xcc09 bytes) 000571 1713364586.431530 mapwrite 0x034e11 thru 0x03ffff (0xb1ef bytes) 000572 1713364586.478091 trunc from 0x040000 to 0x0292b4 (0x16d4b bytes) 000573 1713364586.483970 trunc from 0x0292b5 to 0x01ac79 (0xe63b bytes) 000575 1713364586.489492 write 0x037bac thru 0x03eea4 (0x72f9 bytes) 000576 1713364586.490253 trunc from 0x03eea5 to 0x008169 (0x36d3b bytes) 000578 1713364586.496636 write 0x00870d thru 0x00db31 (0x5425 bytes) 000579 1713364586.498133 mapread 0x009b83 thru 0x00db31 (0x3faf bytes) 000580 1713364586.498694 read 0x004698 thru 0x00db31 (0x949a bytes) 000581 1713364586.499190 mapread 0x00316e thru 0x00db31 (0xa9c4 bytes) 000582 1713364586.499770 mapread 0x000ff7 thru 0x0045d8 (0x35e2 bytes) 000583 1713364586.504050 trunc from 0x00db32 to 0x02ddfd (0x202cc bytes) 000584 1713364586.510809 write 0x03a4a2 thru 0x03ffff (0x5b5e bytes) 000585 1713364586.511650 trunc from 0x040000 to 0x037c0a (0x83f5 bytes) 000586 1713364586.523474 MIRROR_RESYNC 11 mirrors 000589 1713364587.082840 read 0x01c425 thru 0x024194 (0x7d70 bytes) 000592 1713364587.088801 read 0x033c99 thru 0x037c0a (0x3f72 bytes) 000593 1713364587.095861 read 0x02430d thru 0x02dae3 (0x97d7 bytes) 000595 1713364587.096509 read 0x0298bf thru 0x036b54 (0xd296 bytes) 000598 1713364587.097130 mapwrite 0x01a150 thru 0x029513 (0xf3c4 bytes) 000600 1713364587.167645 mapread 0x002241 thru 0x011d0b (0xfacb bytes) 000602 1713364587.168660 read 0x02a6d6 thru 0x035164 (0xaa8f bytes) 000604 1713364587.169384 mapwrite 0x034ebd thru 0x03f1e5 (0xa329 bytes) 000605 1713364587.199292 read 0x020ff8 thru 0x025228 (0x4231 bytes) 000607 1713364587.202595 MIRROR_EXTEND to 12 mirrors 000608 1713364587.403506 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000609 1713364587.508586 read 0x015da6 thru 0x01d19a (0x73f5 bytes) 000612 1713364587.534356 trunc from 0x03f1e6 to 0x013059 (0x2c18c bytes) 000614 1713364587.568673 mapwrite 0x001968 thru 0x00fb24 (0xe1bd bytes) 000615 1713364587.609067 read 0x001177 thru 0x009041 (0x7ecb bytes) 000617 1713364587.610329 write 0x036a85 thru 0x0399bc (0x2f38 bytes) 000618 1713364587.611141 read 0x01ddb0 thru 0x02a63f (0xc890 bytes) 000619 1713364587.616325 mapread 0x004f9e thru 0x00a0b3 (0x5116 bytes) 000620 1713364587.616945 trunc from 0x0399bd to 0x026e7b (0x12b41 bytes) 000621 1713364587.627554 trunc from 0x026e7c to 0x004936 (0x22545 bytes) 000623 1713364587.637295 read 0x0037e6 thru 0x00442d (0xc48 bytes) 000625 1713364587.640058 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000626 1713364587.719046 MIRROR_EXTEND to 11 mirrors 000627 1713364587.916910 write 0x033d65 thru 0x0352c6 (0x1562 bytes) 000628 1713364587.945424 write 0x0114ff thru 0x0159fa (0x44fc bytes) 000629 1713364587.949088 mapwrite 0x02222a thru 0x03186e (0xf645 bytes) 000631 1713364588.011831 mapwrite 0x03104d thru 0x0323f8 (0x13ac bytes) 000633 1713364588.043760 mapwrite 0x0382ce thru 0x03ffff (0x7d32 bytes) 000634 1713364588.093208 write 0x02a2b3 thru 0x02d552 (0x32a0 bytes) 000635 1713364588.095742 MIRROR_EXTEND to 12 mirrors 000636 1713364588.342968 MIRROR_EXTEND to 13 mirrors 000637 1713364588.602507 mapwrite 0x01f81d thru 0x022295 (0x2a79 bytes) 000638 1713364588.699629 read 0x00cd8f thru 0x019fdf (0xd251 bytes) 000639 1713364588.704288 read 0x03dcc6 thru 0x03f235 (0x1570 bytes) 000641 1713364588.709405 mapread 0x027b51 thru 0x0365f0 (0xeaa0 bytes) 000642 1713364588.710254 read 0x00a577 thru 0x010d70 (0x67fa bytes) 000646 1713364588.711364 read 0x01481a thru 0x01bc83 (0x746a bytes) 000647 1713364588.711771 write 0x010dc6 thru 0x01e0e6 (0xd321 bytes) 000648 1713364588.715391 MIRROR_RESYNC 13 mirrors 000649 1713364589.468516 write 0x02290e thru 0x02aa39 (0x812c bytes) 000651 1713364589.515614 mapread 0x021d5d thru 0x02ede0 (0xd084 bytes) 000654 1713364589.526139 trunc from 0x040000 to 0x0076b7 (0x38948 bytes) 000656 1713364589.537157 write 0x01d161 thru 0x02b789 (0xe629 bytes) 000658 1713364589.538666 mapread 0x02ae82 thru 0x02b789 (0x908 bytes) 000659 1713364589.539375 mapread 0x00fe9d thru 0x01b852 (0xb9b6 bytes) 000661 1713364589.550252 trunc from 0x02b78a to 0x021d6d (0x9a1c bytes) 000664 1713364589.560152 mapwrite 0x029d2d thru 0x02e87f (0x4b53 bytes) 000665 1713364589.592875 write 0x02bfd1 thru 0x034acc (0x8afc bytes) 000669 1713364589.594696 mapread 0x022cc6 thru 0x029e4c (0x7187 bytes) 000670 1713364589.595318 write 0x03bd91 thru 0x03ffff (0x426f bytes) 000671 1713364589.598554 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000673 1713364589.685541 trunc from 0x040000 to 0x008c2a (0x373d5 bytes) 000675 1713364589.693044 mapwrite 0x00c29e thru 0x016989 (0xa6ec bytes) 000676 1713364589.737921 mapwrite 0x02bb8a thru 0x02e3c4 (0x283b bytes) 000678 1713364589.778700 read 0x023f2e thru 0x02430f (0x3e2 bytes) 000679 1713364589.782303 write 0x005d06 thru 0x010da4 (0xb09f bytes) 000683 1713364589.791257 mapread 0x002e6a thru 0x00c38a (0x9521 bytes) 000685 1713364589.794145 MIRROR_RESYNC 12 mirrors 000686 1713364590.515094 MIRROR_RESYNC 12 mirrors 000688 1713364590.610983 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000689 1713364590.748106 trunc from 0x02e3c5 to 0x02d691 (0xd33 bytes) 000690 1713364590.786781 mapwrite 0x01760d thru 0x024592 (0xcf86 bytes) 000692 1713364590.827796 write 0x029f56 thru 0x032a4e (0x8af9 bytes) 000693 1713364590.830896 MIRROR_RESYNC 11 mirrors 000694 1713364591.337934 mapwrite 0x035d17 thru 0x03ffff (0xa2e9 bytes) 000699 1713364591.395731 read 0x010c19 thru 0x010d33 (0x11b bytes) 000700 1713364591.400089 mapwrite 0x009036 thru 0x00f5b7 (0x6582 bytes) 000702 1713364591.427681 MIRROR_RESYNC 11 mirrors 000703 1713364591.847158 MIRROR_RESYNC 11 mirrors truncating to largest ever: 0x3e62c 000706 1713364591.940619 trunc from 0x040000 to 0x03e62b (0x19d4 bytes) 000707 1713364591.969575 read 0x03e4a0 thru 0x03e62b (0x18c bytes) 000708 1713364591.971383 write 0x027036 thru 0x033609 (0xc5d4 bytes) 000709 1713364591.976517 MIRROR_EXTEND to 12 mirrors 000711 1713364592.171895 read 0x014ef0 thru 0x021b6b (0xcc7c bytes) 000712 1713364592.177896 read 0x02115e thru 0x02ba41 (0xa8e4 bytes) 000713 1713364592.185111 mapwrite 0x0223ee thru 0x02b000 (0x8c13 bytes) 000715 1713364592.261995 mapread 0x02f7df thru 0x03c3b0 (0xcbd2 bytes) 000716 1713364592.290010 mapwrite 0x03c3b6 thru 0x03ffff (0x3c4a bytes) 000718 1713364592.332222 mapwrite 0x01acf1 thru 0x0211c0 (0x64d0 bytes) 000719 1713364592.389526 MIRROR_RESYNC 12 mirrors 000721 1713364592.841016 trunc from 0x040000 to 0x02c4ec (0x13b13 bytes) 000722 1713364592.883114 trunc from 0x02c4ed to 0x023ead (0x863f bytes) 000723 1713364592.892407 mapwrite 0x01e319 thru 0x029c6a (0xb952 bytes) 000724 1713364592.948209 write 0x0229f8 thru 0x030be9 (0xe1f2 bytes) 000725 1713364592.949224 read 0x01e6dc thru 0x01f977 (0x129c bytes) 000726 1713364592.949644 mapread 0x02dfc4 thru 0x02e05f (0x9c bytes) 000727 1713364592.950153 trunc from 0x030bea to 0x0134f0 (0x1d6f9 bytes) 000728 1713364592.957470 mapwrite 0x01dd70 thru 0x01ff65 (0x21f6 bytes) 000729 1713364592.993762 mapwrite 0x034643 thru 0x03c7e4 (0x81a2 bytes) 000730 1713364593.026761 MIRROR_RESYNC 12 mirrors 000733 1713364593.614164 mapread 0x029674 thru 0x02addf (0x176c bytes) 000734 1713364593.617399 read 0x017bca thru 0x022e53 (0xb28a bytes) 000736 1713364593.620891 write 0x02412f thru 0x0340f6 (0xffc8 bytes) 000738 1713364593.653355 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000739 1713364593.728642 write 0x010b48 thru 0x01635a (0x5813 bytes) 000740 1713364593.734417 mapwrite 0x02270f thru 0x02b1ad (0x8a9f bytes) 000741 1713364593.779417 read 0x0299f0 thru 0x03357b (0x9b8c bytes) 000742 1713364593.780034 read 0x03ad28 thru 0x03c7e4 (0x1abd bytes) 000743 1713364593.784238 mapread 0x022f7f thru 0x026dbd (0x3e3f bytes) 000744 1713364593.784852 write 0x0288c0 thru 0x0326ee (0x9e2f bytes) 000747 1713364593.786358 mapwrite 0x01d00a thru 0x028c13 (0xbc0a bytes) 000748 1713364593.814895 MIRROR_RESYNC 11 mirrors 000749 1713364594.303502 mapread 0x036f92 thru 0x03c7e4 (0x5853 bytes) 000750 1713364594.310935 mapread 0x0141a3 thru 0x01e67e (0xa4dc bytes) 000751 1713364594.317033 mapread 0x0200bc thru 0x020444 (0x389 bytes) 000752 1713364594.317772 mapwrite 0x013cef thru 0x01667d (0x298f bytes) 000753 1713364594.388749 write 0x033024 thru 0x03ffff (0xcfdc bytes) 000754 1713364594.392219 read 0x026ee7 thru 0x02a614 (0x372e bytes) 000755 1713364594.395757 write 0x03cf2f thru 0x03ffff (0x30d1 bytes) 000756 1713364594.398636 MIRROR_RESYNC 11 mirrors 000757 1713364594.821904 write 0x03202c thru 0x0376d4 (0x56a9 bytes) 000758 1713364594.851160 trunc from 0x040000 to 0x011822 (0x2e7dd bytes) 000760 1713364594.865074 read 0x00bc26 thru 0x011822 (0x5bfd bytes) 000761 1713364594.870398 read 0x00ddb6 thru 0x011822 (0x3a6d bytes) 000763 1713364594.871188 trunc from 0x011823 to 0x010ec7 (0x95b bytes) 000764 1713364594.878573 MIRROR_RESYNC 11 mirrors 000765 1713364595.333364 write 0x029ba8 thru 0x034c1d (0xb076 bytes) 000766 1713364595.359336 mapread 0x029f3d thru 0x02f23d (0x5301 bytes) 000767 1713364595.360236 mapread 0x01ae45 thru 0x027455 (0xc611 bytes) 000768 1713364595.373624 MIRROR_RESYNC 11 mirrors 000769 1713364595.845515 mapread 0x020d8b thru 0x02ea26 (0xdc9c bytes) 000771 1713364595.854541 write 0x028356 thru 0x02eed4 (0x6b7f bytes) 000772 1713364595.882274 read 0x020dcd thru 0x0269bd (0x5bf1 bytes) 000775 1713364595.888574 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000776 1713364595.958658 mapread 0x01dc91 thru 0x026222 (0x8592 bytes) 000777 1713364595.965284 trunc from 0x034c1e to 0x000e48 (0x33dd5 bytes) 000779 1713364595.979642 MIRROR_EXTEND to 11 mirrors 000780 1713364596.180945 mapwrite 0x0042a0 thru 0x014093 (0xfdf4 bytes) 000782 1713364596.255074 read 0x00ac83 thru 0x012902 (0x7c80 bytes) 000783 1713364596.255768 write 0x00fe5d thru 0x0110a8 (0x124c bytes) 000789 1713364596.257645 write 0x02edf1 thru 0x03b2c2 (0xc4d2 bytes) 000791 1713364596.260526 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000792 1713364596.339379 read 0x02a7d8 thru 0x032611 (0x7e3a bytes) 000794 1713364596.343207 write 0x0000b0 thru 0x0026ac (0x25fd bytes) 000795 1713364596.346420 mapwrite 0x03fccc thru 0x03ffff (0x334 bytes) 000797 1713364596.392493 trunc from 0x040000 to 0x015061 (0x2af9e bytes) 000798 1713364596.398472 trunc from 0x015062 to 0x03b6f9 (0x26698 bytes) 000799 1713364596.405749 read 0x00b793 thru 0x0150ef (0x995d bytes) 000800 1713364596.410661 read 0x00da63 thru 0x018d2a (0xb2c8 bytes) 000802 1713364596.411464 write 0x0212cd thru 0x025690 (0x43c4 bytes) 000806 1713364596.413042 mapwrite 0x015329 thru 0x0208e8 (0xb5c0 bytes) 000807 1713364596.449661 read 0x03afbf thru 0x03b6f9 (0x73b bytes) 000809 1713364596.450574 write 0x02fd14 thru 0x033ed6 (0x41c3 bytes) 000810 1713364596.451222 mapread 0x012eca thru 0x0185cf (0x5706 bytes) 000811 1713364596.451778 trunc from 0x03b6fa to 0x030a88 (0xac71 bytes) 000812 1713364596.460784 mapwrite 0x0247a5 thru 0x02ffea (0xb846 bytes) 000815 1713364596.504634 mapwrite 0x0243ef thru 0x02ad36 (0x6948 bytes) 000816 1713364596.534471 mapwrite 0x0386c1 thru 0x03ffff (0x793f bytes) 000817 1713364596.580029 MIRROR_EXTEND to 11 mirrors 000818 1713364596.755636 trunc from 0x040000 to 0x015e61 (0x2a19e bytes) 000819 1713364596.797818 read 0x010369 thru 0x015e61 (0x5af9 bytes) 000822 1713364596.801815 write 0x02b53f thru 0x03a7ab (0xf26d bytes) 000823 1713364596.805195 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000824 1713364596.872353 MIRROR_EXTEND to 11 mirrors 000826 1713364597.141476 trunc from 0x03a7ac to 0x01c8d8 (0x1ded3 bytes) 000828 1713364597.173877 mapwrite 0x00a55e thru 0x011fa4 (0x7a47 bytes) 000829 1713364597.206047 mapwrite 0x02b184 thru 0x031c5a (0x6ad7 bytes) 000830 1713364597.255954 write 0x03a638 thru 0x03daa6 (0x346f bytes) 000831 1713364597.256750 write 0x030b72 thru 0x03f322 (0xe7b1 bytes) 000833 1713364597.258089 mapread 0x036941 thru 0x03f322 (0x89e2 bytes) 000834 1713364597.260530 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000835 1713364597.342083 write 0x01437d thru 0x022493 (0xe117 bytes) 000837 1713364597.346596 mapread 0x03cec6 thru 0x03f322 (0x245d bytes) 000838 1713364597.350358 trunc from 0x03f323 to 0x011722 (0x2dc00 bytes) 000839 1713364597.357943 mapwrite 0x01d315 thru 0x01e91b (0x1607 bytes) 000840 1713364597.401871 write 0x03d707 thru 0x03ffff (0x28f9 bytes) 000845 1713364597.404136 mapread 0x00d52d thru 0x011b96 (0x466a bytes) 000847 1713364597.411166 mapwrite 0x030c77 thru 0x03e11a (0xd4a4 bytes) 000849 1713364597.448195 trunc from 0x040000 to 0x00edda (0x31225 bytes) 000850 1713364597.460549 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000852 1713364597.538493 MIRROR_EXTEND to 10 mirrors 000853 1713364597.707976 read 0x003a4a thru 0x00d914 (0x9ecb bytes) 000855 1713364597.716545 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000858 1713364597.831814 trunc from 0x00eddb to 0x01df17 (0xf13d bytes) 000860 1713364597.864660 read 0x007215 thru 0x00bfd3 (0x4dbf bytes) 000861 1713364597.870035 read 0x000c17 thru 0x00b414 (0xa7fe bytes) 000862 1713364597.876686 mapread 0x0109ff thru 0x01df17 (0xd519 bytes) 000863 1713364597.877374 trunc from 0x01df18 to 0x007dd2 (0x16145 bytes) 000866 1713364597.884813 read 0x0076b2 thru 0x007dd2 (0x721 bytes) 000868 1713364597.885568 write 0x012dca thru 0x01b581 (0x87b8 bytes) 000870 1713364597.886921 mapread 0x0140e3 thru 0x01b581 (0x749f bytes) 000872 1713364597.890898 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000873 1713364597.972860 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000874 1713364598.040260 trunc from 0x01b582 to 0x00905f (0x12522 bytes) 000875 1713364598.047117 trunc from 0x009060 to 0x026ba2 (0x1db43 bytes) 000876 1713364598.054179 mapread 0x0022d6 thru 0x00b0c7 (0x8df2 bytes) 000879 1713364598.064118 mapwrite 0x03c5a9 thru 0x03ffff (0x3a57 bytes) 000880 1713364598.106474 mapread 0x01a9b6 thru 0x01b95d (0xfa8 bytes) 000881 1713364598.114690 MIRROR_RESYNC 7 mirrors 000882 1713364598.500250 write 0x016be4 thru 0x0268ab (0xfcc8 bytes) 000884 1713364598.527608 mapread 0x01fafe thru 0x026c39 (0x713c bytes) 000885 1713364598.530213 MIRROR_RESYNC 7 mirrors 000886 1713364598.917010 trunc from 0x040000 to 0x008703 (0x378fc bytes) 000887 1713364598.950026 write 0x029d08 thru 0x03291c (0x8c15 bytes) 000888 1713364598.951105 read 0x0051aa thru 0x00a888 (0x56df bytes) truncating to largest ever: 0x3f02b 000889 1713364598.954611 trunc from 0x03291d to 0x03f02a (0xc70e bytes) 000890 1713364598.962240 read 0x016866 thru 0x01f531 (0x8ccc bytes) 000891 1713364598.967268 read 0x029020 thru 0x038a30 (0xfa11 bytes) 000892 1713364598.967740 read 0x026afc thru 0x02aa27 (0x3f2c bytes) 000893 1713364598.968129 read 0x033405 thru 0x03f02a (0xbc26 bytes) 000896 1713364598.969161 mapwrite 0x000973 thru 0x00d68b (0xcd19 bytes) 000898 1713364599.004654 write 0x03e73f thru 0x03f0a4 (0x966 bytes) 000900 1713364599.006555 mapread 0x016850 thru 0x01b08c (0x483d bytes) 000901 1713364599.007463 mapwrite 0x037278 thru 0x03ffff (0x8d88 bytes) 000902 1713364599.056226 trunc from 0x040000 to 0x00d57a (0x32a85 bytes) 000904 1713364599.062939 trunc from 0x00d57b to 0x03d990 (0x30416 bytes) 000909 1713364599.071875 trunc from 0x03d991 to 0x03142f (0xc561 bytes) 000910 1713364599.077156 mapread 0x03018a thru 0x03142f (0x12a6 bytes) 000913 1713364599.083518 trunc from 0x031430 to 0x039099 (0x7c6a bytes) 000916 1713364599.089943 write 0x02545f thru 0x029d92 (0x4934 bytes) 000918 1713364599.090919 mapwrite 0x00b8a3 thru 0x01196c (0x60ca bytes) 000919 1713364599.125266 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000920 1713364599.207598 trunc from 0x03909a to 0x015b50 (0x23549 bytes) 000921 1713364599.214787 MIRROR_EXTEND to 7 mirrors 000922 1713364599.389524 read 0x010727 thru 0x0108e2 (0x1bc bytes) 000926 1713364599.407542 trunc from 0x015b51 to 0x03d5d5 (0x27a85 bytes) 000928 1713364599.441965 mapread 0x01e680 thru 0x02512c (0x6aad bytes) 000929 1713364599.449296 write 0x02c18f thru 0x038911 (0xc783 bytes) 000932 1713364599.455539 mapread 0x02cc3a thru 0x03bf56 (0xf31d bytes) 000933 1713364599.459486 mapread 0x016434 thru 0x01ebb0 (0x877d bytes) 000935 1713364599.462198 trunc from 0x03d5d6 to 0x015f85 (0x27650 bytes) 000936 1713364599.468227 read 0x00e17f thru 0x015f85 (0x7e07 bytes) 000937 1713364599.471355 mapread 0x00df0f thru 0x015f85 (0x8077 bytes) 000942 1713364599.473838 read 0x0045a8 thru 0x00693d (0x2396 bytes) 000943 1713364599.476128 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000944 1713364599.534327 mapwrite 0x03c857 thru 0x03e0ff (0x18a9 bytes) 000947 1713364599.567631 mapwrite 0x004e0e thru 0x0097a8 (0x499b bytes) 000948 1713364599.597508 read 0x00c9ea thru 0x018b0c (0xc123 bytes) 000950 1713364599.605678 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000951 1713364599.666414 read 0x02064a thru 0x02b794 (0xb14b bytes) 000953 1713364599.672149 mapwrite 0x033c51 thru 0x039334 (0x56e4 bytes) 000954 1713364599.719866 read 0x003644 thru 0x01131d (0xdcda bytes) 000955 1713364599.724643 mapwrite 0x007710 thru 0x00fe96 (0x8787 bytes) 000961 1713364599.750160 write 0x039af0 thru 0x03ffff (0x6510 bytes) 000962 1713364599.752657 MIRROR_RESYNC 5 mirrors 000963 1713364600.012872 mapwrite 0x032c51 thru 0x03c884 (0x9c34 bytes) 000966 1713364600.083714 MIRROR_EXTEND to 6 mirrors 000967 1713364600.267095 read 0x01c59e thru 0x022047 (0x5aaa bytes) 000969 1713364600.274837 MIRROR_EXTEND to 7 mirrors 000971 1713364600.477388 trunc from 0x040000 to 0x03628f (0x9d70 bytes) 000972 1713364600.503002 mapread 0x0192df thru 0x019d5d (0xa7f bytes) 000973 1713364600.505793 mapwrite 0x0149e2 thru 0x024950 (0xff6f bytes) 000976 1713364600.547749 mapwrite 0x0351c9 thru 0x03ffff (0xae37 bytes) 000977 1713364600.593736 mapread 0x026e2a thru 0x031473 (0xa64a bytes) 000981 1713364600.598490 mapread 0x023f22 thru 0x0281a0 (0x427f bytes) 000985 1713364600.601421 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000986 1713364600.673639 mapwrite 0x03140f thru 0x03de71 (0xca63 bytes) 000988 1713364600.715884 MIRROR_EXTEND to 7 mirrors 000993 1713364600.944596 MIRROR_EXTEND to 8 mirrors 000995 1713364601.152208 MIRROR_EXTEND to 9 mirrors 000999 1713364601.368303 mapwrite 0x01a624 thru 0x01eb01 (0x44de bytes) 001000 1713364601.447376 trunc from 0x040000 to 0x0235e9 (0x1ca16 bytes) All operations completed A-OK! PASS 70a (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 10:36:44 (1713364604) Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5539808 .. Extending file size to 1019328 .. Extending file size to 8449280 .. Extending file size to 3053824 .. Extending file size to 3788000 .. Extending file size to 4486880 .. Extending file size to 7035200 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8486048 .. Extending file size to 8660640 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8977664 .. done Extending file size to 8703456 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8972064 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9050944 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9127168 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9113920 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 5157 5158 5160 5161 5163 done 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg450-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:37:51 (1713364671) Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4392000 .. Extending file size to 3547168 .. failed Extending file size to 8715232 .. Extending file size to 8058656 .. Extending file size to 8985120 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9241248 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9041120 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9151232 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 7802 7803 7805 7806 7809 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg450-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:38:57 (1713364737) Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0305023 s, 68.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0667264 s, 31.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:39:03 (1713364743) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0228886 s, 91.6 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 10:39:06 (1713364746) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xab5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xab5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xab6:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xab0:0x0] } PASS 203 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:39:09 (1713364749) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0212037 s, 98.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xab7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:39:12 (1713364752) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0291776 s, 71.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xab8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xab9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xab8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xab9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab4:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:39:15 (1713364755) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0285116 s, 73.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab6:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:39:40 (1713364780) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0293242 s, 35.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xabb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xabb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xabc:0x0] } PASS 204d (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:40:22 (1713364822) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0348361 s, 60.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xabd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xab9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xaba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xabd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:41:08 (1713364868) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. Creating new pool oleg450-server: Pool lustre.test_204f created Adding targets to pool oleg450-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg450-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0255489 s, 82.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xabb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xabe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xabb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xabe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg450-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg450-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg450-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (50s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 10:41:59 (1713364919) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:42:03 (1713364923) oleg450-server: Pool lustre.test_206 created Adding targets to pool oleg450-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg450-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg450-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg450-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg450-server: Pool lustre.test_206 destroyed PASS 206 (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:42:15 (1713364935) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172533 s, 60.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00824805 s, 127 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x697f0adc Mirror 3: 0x697f0adc PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:42:19 (1713364939) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:42:21 (1713364941) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:42:23 (1713364943) Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0163011 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1470000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1470000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1470000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xfc0000 Read this (4 bytes): BBBB 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:42:39 (1713364959) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00262575 s, 3.1 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 10:42:42 (1713364962) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113873 s, 92.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5f9 lmm_fid: [0x200000404:0x5f9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5f9 lmm_fid: [0x200000404:0x5f9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xac4:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835449 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-17 10:42:42.000000000 -0400 Modify: 2024-04-17 10:42:42.000000000 -0400 Change: 2024-04-17 10:42:43.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:42:46 (1713364966) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 787 sec ============ 10:42:47 (1713364967) === sanity-flr: start cleanup 10:42:47 (1713364967) === === sanity-flr: finish cleanup 10:42:48 (1713364968) ===