-----============= acceptance-small: sanity-flr ============----- Thu Apr 18 04:40:08 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:40:13 (1713429613) === oleg127-client.virtnet: executing check_config_client /mnt/lustre oleg127-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg127-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b39be800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b39be800.idle_timeout=debug disable quota as required oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:40:21 (1713429621) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6909] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 04:40:23 (1713429623) PASS 0a (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:40:31 (1713429631) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:40:35 (1713429635) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 04:40:39 (1713429639) PASS 0d (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:40:48 (1713429648) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:40:51 (1713429651) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 04:40:55 (1713429655) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b39be800.stats=clear osc.lustre-OST0001-osc-ffff8800b39be800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:41:00 (1713429660) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:41:06 (1713429666) PASS 0j (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:41:11 (1713429671) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 04:41:15 (1713429675) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 04:41:21 (1713429681) PASS 2 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:41:26 (1713429686) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:41:30 (1713429690) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:41:34 (1713429694) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 04:41:38 (1713429698) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:41:42 (1713429702) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0781859 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0598034 s, 17.5 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:41:46 (1713429706) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.0893482 s, 164 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:41:50 (1713429710) Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b39be800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b39be800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b39be800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b39be800.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:42:18 (1713429738) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.159329 s, 65.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b39be800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b39be800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:42:33 (1713429753) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b39be800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b39be800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b39be800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b39be800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:42:59 (1713429779) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.939703 s, 55.8 MB/s -rw-r--r-- 1 root root 52428800 Apr 18 04:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.334s user 0m0.000s sys 0m0.124s Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b39be800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b39be800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.282s user 0m0.000s sys 0m0.124s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b39be800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b39be800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.280s user 0m0.000s sys 0m0.122s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b39be800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b39be800.ost_server_uuid in FULL state after 0 sec PASS 33b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:43:25 (1713429805) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:43:28 (1713429808) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:43:31 (1713429811) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 04:43:34 (1713429814) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 04:43:39 (1713429819) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.185822 s, 67.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:43:45 (1713429825) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00972481 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104591 s, 100 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:43:49 (1713429829) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:43:55 (1713429835) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000919818 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.001064 s, 962 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000695 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00126795 s, 808 kB/s flv=10 olv=10 PASS 36d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 04:44:00 (1713429840) create test files with size 15740369 .. 6143a8a512399531928da802470c7e1b - 4508a5a9b96fdd395b259c95811de31a - a6dac0d664bf3989e7b45b10c6ed5f0d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre PASS 37 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 04:44:09 (1713429849) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:44:16 (1713429856) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:44:19 (1713429859) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026722 s, 39.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4c:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108357 s, 96.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 04:44:24 (1713429864) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0499836 s, 42.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.121476 s, 34.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0668534 s, 31.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 04:44:30 (1713429870) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 04:44:33 (1713429873) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:44:36 (1713429876) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01489 s, 70.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 04:44:40 (1713429880) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:44:43 (1713429883) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x55:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:44:48 (1713429888) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0705008 s, 149 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:44:53 (1713429893) llite.lustre-ffff8800b39b9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0890748 s, 118 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012ab27800.xattr_cache=1 PASS 44e (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:44:59 (1713429899) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (9s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 04:45:10 (1713429910) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 04:45:19 (1713429919) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 04:45:22 (1713429922) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0837455 s, 37.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0548939 s, 57.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: fa940eedbc2433bbdb72c5c6fe71afd2 - mirror 1 checksum: 12a6df346d982ae0d7a09635920971a1 - mirror 2 checksum: fa940eedbc2433bbdb72c5c6fe71afd2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 04:45:22 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 18 04:45:22 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fa940eedbc2433bbdb72c5c6fe71afd2 - mirror 1 checksum: 12a6df346d982ae0d7a09635920971a1 - mirror 2 checksum: 12a6df346d982ae0d7a09635920971a1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 04:45:22 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 18 04:45:22 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 48 (3s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:45:27 (1713429927) Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 04:45:29 (1713429929) shut down Failover mds1 to oleg127-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 04:45:42 (1713429942) targets are mounted 04:45:42 (1713429942) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011909 s, 88.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 50A (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:45:49 (1713429949) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 04:45:52 (1713429952) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:45:55 (1713429955) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00274082 s, 6.0 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 04:45:59 (1713429959) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:46:01 (1713429961) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:46:04 (1713429964) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:46:11 (1713429971) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:46:31 (1713429991) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:46:49 (1713430009) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:47:07 (1713430027) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 04:47:10 (1713430030) mirror create pid 26722 mirror split pid 26723 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26722 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26723 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 04:48:14 (1713430094) Seed set to 94 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2a55d 000003 1713430094.442873 trunc from 00000000 to 0x02a55c (0x2a55d bytes) 000005 1713430094.446153 mapread 0x01870f thru 0x027ae9 (0xf3db bytes) 000008 1713430094.452806 mapread 0x0086b7 thru 0x014bb7 (0xc501 bytes) 000010 1713430094.457003 mapwrite 0x003445 thru 0x0083b4 (0x4f70 bytes) 000011 1713430094.482423 mapwrite 0x008e2e thru 0x010d77 (0x7f4a bytes) 000012 1713430094.496627 mapread 0x02662c thru 0x02a55c (0x3f31 bytes) 000013 1713430094.497193 mapwrite 0x013923 thru 0x019884 (0x5f62 bytes) 000014 1713430094.509969 mapwrite 0x0347c5 thru 0x037c0f (0x344b bytes) 000015 1713430094.528594 write 0x00126f thru 0x0023f6 (0x1188 bytes) 000017 1713430094.529180 mapwrite 0x02cd2b thru 0x03649d (0x9773 bytes) 000021 1713430094.546071 read 0x028348 thru 0x030250 (0x7f09 bytes) 000022 1713430094.546382 mapwrite 0x02071d thru 0x025068 (0x494c bytes) 000023 1713430094.560511 trunc from 0x037c10 to 0x020c66 (0x16fa9 bytes) 000026 1713430094.564742 mapread 0x01eca9 thru 0x020c66 (0x1fbe bytes) 000027 1713430094.565123 mapread 0x016e39 thru 0x01ff8b (0x9153 bytes) 000028 1713430094.566690 MIRROR_EXTEND to 2 mirrors 000030 1713430094.666826 write 0x037285 thru 0x03ffff (0x8d7b bytes) 000031 1713430094.682818 mapwrite 0x011b70 thru 0x0151f0 (0x3681 bytes) 000033 1713430094.705661 mapread 0x02d09d thru 0x03033c (0x32a0 bytes) 000035 1713430094.709071 write 0x000c50 thru 0x00dd08 (0xd0b9 bytes) 000036 1713430094.712133 write 0x00c5f0 thru 0x016f02 (0xa913 bytes) 000037 1713430094.712995 mapwrite 0x00f437 thru 0x01667f (0x7249 bytes) 000038 1713430094.729509 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3384c 000039 1713430094.828282 trunc from 0x040000 to 0x03384b (0xc7b4 bytes) 000040 1713430094.854597 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000042 1713430094.893099 read 0x029591 thru 0x03384b (0xa2bb bytes) 000043 1713430094.898098 read 0x001e0f thru 0x00d99a (0xbb8c bytes) 000044 1713430094.904175 mapwrite 0x00da6f thru 0x014f48 (0x74da bytes) 000046 1713430094.934302 trunc from 0x03384c to 0x0014de (0x3236d bytes) 000048 1713430094.941044 mapread 0x000a1c thru 0x0014de (0xac3 bytes) 000050 1713430094.942084 write 0x00ff04 thru 0x0186b5 (0x87b2 bytes) 000051 1713430094.942727 write 0x005d6c thru 0x013765 (0xd9fa bytes) 000052 1713430094.945388 trunc from 0x0186b6 to 0x0206f0 (0x803b bytes) 000054 1713430094.950803 mapread 0x01af74 thru 0x0206f0 (0x577d bytes) 000058 1713430094.959857 trunc from 0x0206f1 to 0x01882c (0x7ec4 bytes) 000060 1713430094.963858 read 0x0061b7 thru 0x00c32f (0x6179 bytes) 000061 1713430094.964222 trunc from 0x01882d to 0x01706e (0x17be bytes) 000063 1713430094.968705 write 0x02ec7a thru 0x030777 (0x1afe bytes) 000066 1713430094.969999 write 0x039f3c thru 0x03e299 (0x435e bytes) truncating to largest ever: 0x3637f 000067 1713430094.970736 trunc from 0x03e29a to 0x03637e (0x7f1b bytes) 000074 1713430094.976732 mapwrite 0x03c838 thru 0x03ffff (0x37c8 bytes) 000077 1713430095.001474 mapread 0x012841 thru 0x01c2ea (0x9aaa bytes) 000078 1713430095.001874 mapread 0x0087a8 thru 0x014337 (0xbb90 bytes) 000079 1713430095.002318 trunc from 0x040000 to 0x0029d2 (0x3d62d bytes) 000080 1713430095.005878 trunc from 0x0029d3 to 0x034690 (0x31cbe bytes) 000081 1713430095.008684 read 0x02eae6 thru 0x02eeb5 (0x3d0 bytes) 000084 1713430095.012666 read 0x00ad28 thru 0x00dd06 (0x2fdf bytes) 000085 1713430095.012900 trunc from 0x034691 to 0x02199f (0x12cf1 bytes) 000087 1713430095.016371 mapread 0x00cd6f thru 0x01158e (0x4820 bytes) 000088 1713430095.016746 write 0x0267ab thru 0x02d2a6 (0x6afc bytes) 000092 1713430095.019318 MIRROR_EXTEND to 2 mirrors 000094 1713430095.126403 trunc from 0x02d2a7 to 0x035e8e (0x8be8 bytes) 000096 1713430095.144164 trunc from 0x035e8f to 0x002e7a (0x33014 bytes) 000097 1713430095.150015 MIRROR_EXTEND to 3 mirrors 000098 1713430095.246654 mapwrite 0x00c511 thru 0x01b8af (0xf39f bytes) 000099 1713430095.285729 MIRROR_RESYNC 3 mirrors 000100 1713430095.392544 read 0x0078eb thru 0x0124c7 (0xabdd bytes) truncating to largest ever: 0x37779 000101 1713430095.394732 trunc from 0x01b8b0 to 0x037778 (0x1bec9 bytes) 000102 1713430095.412534 mapwrite 0x008372 thru 0x00dfc4 (0x5c53 bytes) 000104 1713430095.434165 mapwrite 0x03f2e5 thru 0x03ffff (0xd1b bytes) 000105 1713430095.464152 mapwrite 0x01aa14 thru 0x01e35a (0x3947 bytes) 000106 1713430095.484490 read 0x020894 thru 0x028d24 (0x8491 bytes) 000108 1713430095.488914 write 0x00d142 thru 0x0161a3 (0x9062 bytes) 000109 1713430095.489696 mapread 0x003f41 thru 0x00e303 (0xa3c3 bytes) 000113 1713430095.490915 write 0x012a75 thru 0x015b76 (0x3102 bytes) 000114 1713430095.491485 trunc from 0x040000 to 0x01e833 (0x217cc bytes) 000117 1713430095.497848 mapwrite 0x01f64c thru 0x02a957 (0xb30c bytes) 000119 1713430095.534592 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000123 1713430095.580182 trunc from 0x02a958 to 0x0345e1 (0x9c8a bytes) 000124 1713430095.584007 trunc from 0x0345e2 to 0x02db41 (0x6aa0 bytes) 000125 1713430095.589442 MIRROR_RESYNC 2 mirrors 000129 1713430095.674987 read 0x01d2f3 thru 0x024457 (0x7165 bytes) 000130 1713430095.677368 trunc from 0x02db42 to 0x014d51 (0x18df0 bytes) 000134 1713430095.699392 write 0x013897 thru 0x017470 (0x3bda bytes) 000137 1713430095.702135 read 0x00c235 thru 0x0172c5 (0xb091 bytes) 000138 1713430095.705161 trunc from 0x017471 to 0x015211 (0x225f bytes) 000139 1713430095.709937 write 0x016897 thru 0x01c9c3 (0x612d bytes) 000140 1713430095.710526 write 0x01d670 thru 0x01dc0c (0x59d bytes) 000142 1713430095.711211 mapwrite 0x0138cb thru 0x014306 (0xa3c bytes) 000143 1713430095.728590 mapwrite 0x03f907 thru 0x03ffff (0x6f9 bytes) 000146 1713430095.750174 mapwrite 0x035d88 thru 0x038a12 (0x2c8b bytes) 000147 1713430095.764363 write 0x026091 thru 0x032f87 (0xcef7 bytes) 000150 1713430095.765521 write 0x03b129 thru 0x03ffff (0x4ed7 bytes) 000151 1713430095.767623 MIRROR_EXTEND to 3 mirrors 000154 1713430095.887730 write 0x019b30 thru 0x024fa9 (0xb47a bytes) 000155 1713430095.906408 MIRROR_RESYNC 3 mirrors 000156 1713430096.027991 MIRROR_EXTEND to 4 mirrors 000159 1713430096.154182 read 0x030f2e thru 0x03cff5 (0xc0c8 bytes) 000163 1713430096.156620 mapwrite 0x0015da thru 0x002d30 (0x1757 bytes) 000164 1713430096.206115 read 0x0244f0 thru 0x025f34 (0x1a45 bytes) 000165 1713430096.211027 trunc from 0x040000 to 0x00a779 (0x35886 bytes) 000167 1713430096.222142 trunc from 0x00a77a to 0x01a89e (0x10125 bytes) 000169 1713430096.229573 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000171 1713430096.276000 read 0x00dab6 thru 0x010d79 (0x32c4 bytes) 000173 1713430096.278936 MIRROR_EXTEND to 4 mirrors 000179 1713430096.386114 mapwrite 0x017130 thru 0x022db6 (0xbc87 bytes) 000181 1713430096.430115 read 0x01b13f thru 0x0212d9 (0x619b bytes) 000182 1713430096.430533 read 0x010f3b thru 0x0112cd (0x393 bytes) 000184 1713430096.433977 trunc from 0x022db7 to 0x0181b9 (0xabfd bytes) 000187 1713430096.440129 trunc from 0x0181ba to 0x00db79 (0xa640 bytes) 000188 1713430096.444353 mapwrite 0x01a1f2 thru 0x025efc (0xbd0b bytes) 000189 1713430096.474859 trunc from 0x025efd to 0x031bf6 (0xbcfa bytes) 000191 1713430096.479035 trunc from 0x031bf7 to 0x031ff2 (0x3fc bytes) 000194 1713430096.483359 mapwrite 0x03a4b1 thru 0x03ffff (0x5b4f bytes) 000196 1713430096.505365 trunc from 0x040000 to 0x020706 (0x1f8f9 bytes) 000197 1713430096.509302 write 0x03d27e thru 0x03ffff (0x2d82 bytes) 000202 1713430096.511245 read 0x02744e thru 0x03182d (0xa3e0 bytes) 000206 1713430096.516451 trunc from 0x040000 to 0x01f8be (0x20741 bytes) 000208 1713430096.520652 read 0x0129a0 thru 0x012b00 (0x161 bytes) 000213 1713430096.522074 write 0x01dcbe thru 0x0254e2 (0x7825 bytes) 000214 1713430096.524350 MIRROR_EXTEND to 5 mirrors 000215 1713430096.646399 mapwrite 0x03031f thru 0x0350eb (0x4dcd bytes) 000220 1713430096.693181 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000221 1713430096.746715 trunc from 0x0350ec to 0x035729 (0x63e bytes) 000222 1713430096.752310 read 0x024987 thru 0x02ce48 (0x84c2 bytes) 000223 1713430096.755819 read 0x0258cf thru 0x0316c3 (0xbdf5 bytes) 000228 1713430096.762014 mapwrite 0x029a75 thru 0x0397e9 (0xfd75 bytes) 000229 1713430096.793244 mapread 0x029072 thru 0x0340e8 (0xb077 bytes) 000230 1713430096.793811 mapread 0x000d45 thru 0x010574 (0xf830 bytes) 000231 1713430096.795949 MIRROR_EXTEND to 5 mirrors 000232 1713430096.923363 write 0x026334 thru 0x03632e (0xfffb bytes) 000233 1713430096.942362 trunc from 0x0397ea to 0x02aa02 (0xede7 bytes) 000234 1713430096.954097 mapread 0x00e218 thru 0x01397d (0x5766 bytes) truncating to largest ever: 0x3e5ac 000235 1713430096.958945 trunc from 0x02aa03 to 0x03e5ab (0x13ba9 bytes) 000237 1713430096.964791 mapwrite 0x0270fb thru 0x02afdd (0x3ee3 bytes) 000238 1713430096.987884 mapwrite 0x02e134 thru 0x02e2b7 (0x184 bytes) 000239 1713430097.005071 mapread 0x037b81 thru 0x03e5ab (0x6a2b bytes) 000240 1713430097.008160 write 0x0256fe thru 0x02d17e (0x7a81 bytes) 000241 1713430097.011146 mapread 0x024533 thru 0x02e1e0 (0x9cae bytes) 000242 1713430097.011724 trunc from 0x03e5ac to 0x001d88 (0x3c823 bytes) 000243 1713430097.016000 read 0x000f70 thru 0x001d88 (0xe19 bytes) 000244 1713430097.017679 trunc from 0x001d89 to 0x02fefb (0x2e173 bytes) 000245 1713430097.021469 trunc from 0x02fefc to 0x031744 (0x1849 bytes) 000247 1713430097.025267 read 0x027ded thru 0x02dcce (0x5ee2 bytes) 000248 1713430097.030013 mapread 0x030830 thru 0x031744 (0xf15 bytes) 000251 1713430097.032701 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000253 1713430097.089925 MIRROR_EXTEND to 5 mirrors 000254 1713430097.219138 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000256 1713430097.288878 mapwrite 0x00142a thru 0x00e1a2 (0xcd79 bytes) 000257 1713430097.350378 write 0x028e21 thru 0x03531c (0xc4fc bytes) 000258 1713430097.355096 mapread 0x006e46 thru 0x016911 (0xfacc bytes) 000262 1713430097.360484 mapwrite 0x033917 thru 0x03e053 (0xa73d bytes) 000264 1713430097.397315 write 0x0162c2 thru 0x01f094 (0x8dd3 bytes) 000265 1713430097.399516 write 0x02f988 thru 0x03bc7c (0xc2f5 bytes) 000266 1713430097.400285 mapwrite 0x00b625 thru 0x0108d2 (0x52ae bytes) 000269 1713430097.424001 trunc from 0x03e054 to 0x010a29 (0x2d62a bytes) 000270 1713430097.430772 MIRROR_RESYNC 4 mirrors 000275 1713430097.630201 read 0x007f8c thru 0x010a29 (0x8a9e bytes) 000277 1713430097.637099 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000278 1713430097.707632 MIRROR_EXTEND to 4 mirrors 000279 1713430097.830504 mapwrite 0x027ed9 thru 0x02efc8 (0x70f0 bytes) 000281 1713430097.883338 mapwrite 0x030ba6 thru 0x031908 (0xd63 bytes) 000282 1713430097.913617 mapread 0x015ece thru 0x01afff (0x5132 bytes) 000284 1713430097.917693 write 0x02e726 thru 0x0360de (0x79b9 bytes) 000289 1713430097.921659 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000290 1713430097.979041 mapread 0x017340 thru 0x0212a3 (0x9f64 bytes) 000292 1713430097.987976 mapwrite 0x023582 thru 0x031bce (0xe64d bytes) 000294 1713430098.010599 trunc from 0x0360df to 0x029ac2 (0xc61c bytes) 000295 1713430098.014595 mapread 0x00b356 thru 0x010669 (0x5314 bytes) 000298 1713430098.017321 trunc from 0x029ac3 to 0x03cee1 (0x1341f bytes) 000299 1713430098.020855 mapread 0x02ce89 thru 0x0315fd (0x4775 bytes) 000300 1713430098.023596 trunc from 0x03cee2 to 0x02950c (0x139d5 bytes) 000301 1713430098.027774 mapread 0x007d8f thru 0x00d09e (0x5310 bytes) 000303 1713430098.030783 mapwrite 0x03bcc6 thru 0x03f5ff (0x393a bytes) 000304 1713430098.057764 read 0x00d5a4 thru 0x01c3bc (0xee19 bytes) 000305 1713430098.058353 trunc from 0x03f600 to 0x000c0c (0x3e9f3 bytes) 000306 1713430098.062968 trunc from 0x000c0d to 0x00676b (0x5b5f bytes) 000308 1713430098.066720 write 0x02450e thru 0x028045 (0x3b38 bytes) 000312 1713430098.068279 write 0x024cfd thru 0x02dd8e (0x9092 bytes) 000313 1713430098.069025 mapread 0x017004 thru 0x01ed2f (0x7d2c bytes) 000316 1713430098.073100 mapwrite 0x00320d thru 0x00b672 (0x8466 bytes) 000317 1713430098.097604 MIRROR_EXTEND to 4 mirrors 000318 1713430098.228677 MIRROR_RESYNC 4 mirrors 000322 1713430098.360638 read 0x019976 thru 0x01e83c (0x4ec7 bytes) 000323 1713430098.363969 MIRROR_RESYNC 4 mirrors 000324 1713430098.410402 mapwrite 0x003270 thru 0x010666 (0xd3f7 bytes) 000325 1713430098.463087 trunc from 0x02dd8f to 0x01e8a9 (0xf4e5 bytes) 000326 1713430098.467241 write 0x02be91 thru 0x02f9a2 (0x3b12 bytes) 000327 1713430098.467901 read 0x018a50 thru 0x0192eb (0x89c bytes) 000328 1713430098.470671 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000329 1713430098.515710 write 0x030882 thru 0x031739 (0xeb8 bytes) 000331 1713430098.516617 read 0x00a24b thru 0x0118b7 (0x766d bytes) 000334 1713430098.520651 trunc from 0x03173a to 0x011f2c (0x1f80d bytes) 000335 1713430098.524653 mapread 0x00f2c0 thru 0x011f2c (0x2c6d bytes) 000336 1713430098.525276 mapwrite 0x00ede4 thru 0x01cb56 (0xdd73 bytes) 000337 1713430098.550943 mapread 0x006a85 thru 0x00f232 (0x87ae bytes) 000338 1713430098.553418 write 0x01b4f7 thru 0x029048 (0xdb52 bytes) 000339 1713430098.555627 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000340 1713430098.601211 mapwrite 0x036505 thru 0x03b074 (0x4b70 bytes) 000343 1713430098.628838 trunc from 0x03b075 to 0x03b5f1 (0x57d bytes) 000345 1713430098.632580 mapwrite 0x000566 thru 0x00e7df (0xe27a bytes) 000349 1713430098.652768 mapwrite 0x03aa19 thru 0x03ffff (0x55e7 bytes) 000350 1713430098.673525 trunc from 0x040000 to 0x006a0b (0x395f4 bytes) 000351 1713430098.678736 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000352 1713430098.704815 read 0x004037 thru 0x006a0b (0x29d5 bytes) 000356 1713430098.707768 mapwrite 0x02c3d5 thru 0x0312c8 (0x4ef4 bytes) 000362 1713430098.731869 read 0x02dd8d thru 0x0312c8 (0x353c bytes) 000363 1713430098.732251 write 0x011ea3 thru 0x013eb0 (0x200e bytes) 000364 1713430098.736095 trunc from 0x0312c9 to 0x037ac0 (0x67f8 bytes) 000365 1713430098.739463 trunc from 0x037ac1 to 0x0078f5 (0x301cb bytes) 000368 1713430098.743493 mapread 0x0018d9 thru 0x007884 (0x5fac bytes) 000370 1713430098.744150 mapread 0x000657 thru 0x0078f5 (0x729f bytes) 000371 1713430098.744480 write 0x01b952 thru 0x026958 (0xb007 bytes) 000373 1713430098.745427 trunc from 0x026959 to 0x029769 (0x2e11 bytes) 000374 1713430098.748617 mapread 0x010236 thru 0x01d4ae (0xd279 bytes) 000375 1713430098.759415 read 0x016b01 thru 0x01b4ba (0x49ba bytes) 000377 1713430098.760098 mapwrite 0x03f9a2 thru 0x03ffff (0x65e bytes) 000380 1713430098.780647 trunc from 0x040000 to 0x019c83 (0x2637c bytes) 000386 1713430098.785244 mapwrite 0x036178 thru 0x03816e (0x1ff7 bytes) 000387 1713430098.804136 mapwrite 0x024f97 thru 0x02afa3 (0x600d bytes) 000392 1713430098.825115 mapwrite 0x0212fe thru 0x029afa (0x87fd bytes) 000393 1713430098.842234 write 0x018c09 thru 0x01dff0 (0x53e8 bytes) 000394 1713430098.844075 read 0x0345a9 thru 0x03816e (0x3bc6 bytes) 000395 1713430098.846450 mapwrite 0x01be24 thru 0x01d4a2 (0x167f bytes) 000397 1713430098.860391 mapwrite 0x03ea44 thru 0x03ffff (0x15bc bytes) 000400 1713430098.881597 MIRROR_EXTEND to 2 mirrors 000401 1713430098.996798 read 0x031373 thru 0x03509f (0x3d2d bytes) 000402 1713430098.999676 mapread 0x01e8b7 thru 0x02a56c (0xbcb6 bytes) 000403 1713430099.006924 mapread 0x02c55c thru 0x035bc1 (0x9666 bytes) 000404 1713430099.009128 write 0x029b9c thru 0x02ef3e (0x53a3 bytes) 000405 1713430099.029707 trunc from 0x040000 to 0x018745 (0x278ba bytes) 000406 1713430099.038748 write 0x03351a thru 0x034d4b (0x1832 bytes) 000407 1713430099.039368 read 0x027178 thru 0x02d463 (0x62ec bytes) 000408 1713430099.045051 MIRROR_RESYNC 2 mirrors 000410 1713430099.152444 write 0x03e7d2 thru 0x03ffff (0x182e bytes) 000411 1713430099.175170 MIRROR_RESYNC 2 mirrors 000414 1713430099.298835 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000416 1713430099.334944 mapread 0x01eaad thru 0x023916 (0x4e6a bytes) 000417 1713430099.342621 write 0x02e4fc thru 0x03c495 (0xdf9a bytes) 000418 1713430099.350368 trunc from 0x040000 to 0x022001 (0x1dffe bytes) 000419 1713430099.359387 mapread 0x01deb3 thru 0x022001 (0x414f bytes) 000421 1713430099.363163 trunc from 0x022002 to 0x007328 (0x1acd9 bytes) 000422 1713430099.367049 read 0x0028fd thru 0x007328 (0x4a2c bytes) 000423 1713430099.369272 write 0x01868d thru 0x021ef5 (0x9869 bytes) 000424 1713430099.370053 mapwrite 0x03f01b thru 0x03ffff (0xfe5 bytes) 000426 1713430099.393973 mapwrite 0x01ecb5 thru 0x021e5c (0x31a8 bytes) 000427 1713430099.411014 read 0x0112bd thru 0x01168b (0x3cf bytes) 000428 1713430099.415239 mapread 0x0147e9 thru 0x014b8c (0x3a4 bytes) 000430 1713430099.417704 MIRROR_EXTEND to 2 mirrors 000432 1713430099.539784 mapread 0x0308a4 thru 0x034aec (0x4249 bytes) 000433 1713430099.545428 mapread 0x01c962 thru 0x02934c (0xc9eb bytes) 000434 1713430099.552840 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000436 1713430099.586410 trunc from 0x040000 to 0x009079 (0x36f86 bytes) 000437 1713430099.595794 mapread 0x006099 thru 0x009079 (0x2fe1 bytes) 000439 1713430099.600481 write 0x03d4a5 thru 0x03ffff (0x2b5b bytes) 000440 1713430099.601262 mapwrite 0x01f497 thru 0x026b72 (0x76dc bytes) 000441 1713430099.619873 trunc from 0x040000 to 0x02817f (0x17e80 bytes) 000445 1713430099.625250 write 0x00d954 thru 0x01bfcf (0xe67c bytes) 000447 1713430099.629291 mapwrite 0x015453 thru 0x01adc8 (0x5976 bytes) 000448 1713430099.647509 write 0x00e1c5 thru 0x015968 (0x77a4 bytes) 000450 1713430099.648701 mapwrite 0x00081b thru 0x00ff94 (0xf77a bytes) 000451 1713430099.674645 read 0x01d276 thru 0x02817f (0xaf0a bytes) 000455 1713430099.678052 write 0x023ed0 thru 0x02828e (0x43bf bytes) 000456 1713430099.678704 trunc from 0x02828f to 0x0141e8 (0x140a6 bytes) 000459 1713430099.683589 read 0x00c069 thru 0x0141e8 (0x8180 bytes) 000460 1713430099.683952 read 0x001238 thru 0x00bc78 (0xaa41 bytes) 000464 1713430099.684769 mapwrite 0x03fd6b thru 0x03ffff (0x295 bytes) 000465 1713430099.708456 mapread 0x013569 thru 0x01c56c (0x9004 bytes) 000468 1713430099.709361 mapwrite 0x038044 thru 0x039e6c (0x1e29 bytes) 000469 1713430099.724056 read 0x01af15 thru 0x01e785 (0x3871 bytes) 000470 1713430099.724343 read 0x011f1e thru 0x01ebd2 (0xccb5 bytes) 000471 1713430099.724627 mapwrite 0x030e84 thru 0x03f218 (0xe395 bytes) 000476 1713430099.740542 mapwrite 0x01722f thru 0x02631e (0xf0f0 bytes) 000478 1713430099.756221 mapwrite 0x01cc69 thru 0x029852 (0xcbea bytes) 000479 1713430099.772221 trunc from 0x040000 to 0x013c56 (0x2c3a9 bytes) 000483 1713430099.776812 mapread 0x0031b8 thru 0x00d7f4 (0xa63d bytes) 000484 1713430099.777385 mapread 0x006c7f thru 0x00851f (0x18a1 bytes) 000485 1713430099.777883 write 0x039590 thru 0x03ffff (0x6a70 bytes) 000486 1713430099.778591 write 0x001d13 thru 0x00d039 (0xb327 bytes) 000487 1713430099.779078 write 0x03b557 thru 0x03d5ff (0x20a9 bytes) 000488 1713430099.779495 mapread 0x0049b1 thru 0x0102b3 (0xb903 bytes) 000493 1713430099.781113 mapwrite 0x00aefe thru 0x014d97 (0x9e9a bytes) 000494 1713430099.802508 mapread 0x031fc8 thru 0x0330e1 (0x111a bytes) 000495 1713430099.803082 mapwrite 0x00cc54 thru 0x01a7f8 (0xdba5 bytes) 000496 1713430099.819647 read 0x00f95d thru 0x014677 (0x4d1b bytes) 000498 1713430099.820917 mapwrite 0x03f701 thru 0x03ffff (0x8ff bytes) 000499 1713430099.832889 read 0x006275 thru 0x00855b (0x22e7 bytes) 000500 1713430099.834693 MIRROR_EXTEND to 2 mirrors 000501 1713430099.961780 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000503 1713430099.994305 write 0x034d42 thru 0x03cd31 (0x7ff0 bytes) 000504 1713430099.999392 read 0x01adbe thru 0x01d336 (0x2579 bytes) 000506 1713430100.001972 mapwrite 0x00301a thru 0x0104bb (0xd4a2 bytes) 000509 1713430100.026369 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3eaca 000510 1713430100.150679 trunc from 0x040000 to 0x03eac9 (0x1536 bytes) 000511 1713430100.172118 mapread 0x0333d4 thru 0x03e1dd (0xae0a bytes) 000513 1713430100.179605 mapread 0x0255b6 thru 0x0350b6 (0xfb01 bytes) 000516 1713430100.182841 mapread 0x0146ca thru 0x01aad7 (0x640e bytes) 000518 1713430100.187812 MIRROR_RESYNC 2 mirrors 000520 1713430100.294109 write 0x00eaa8 thru 0x01a849 (0xbda2 bytes) 000522 1713430100.309830 trunc from 0x03eaca to 0x005eb9 (0x38c10 bytes) 000526 1713430100.319044 mapwrite 0x00d1d1 thru 0x01aac7 (0xd8f7 bytes) 000527 1713430100.351537 read 0x0165e5 thru 0x01aac7 (0x44e3 bytes) 000532 1713430100.353433 mapread 0x017a17 thru 0x01aac7 (0x30b1 bytes) 000533 1713430100.354184 read 0x01026d thru 0x0159a4 (0x5738 bytes) 000534 1713430100.354587 mapwrite 0x029ae5 thru 0x02b895 (0x1db1 bytes) 000535 1713430100.385634 MIRROR_RESYNC 2 mirrors 000537 1713430100.498831 mapread 0x021767 thru 0x02b895 (0xa12f bytes) 000538 1713430100.503495 read 0x01810c thru 0x0239c7 (0xb8bc bytes) 000540 1713430100.506000 write 0x02d51e thru 0x034589 (0x706c bytes) 000541 1713430100.518550 trunc from 0x03458a to 0x01d0d2 (0x174b7 bytes) 000542 1713430100.526730 mapwrite 0x014432 thru 0x018909 (0x44d8 bytes) 000544 1713430100.549684 mapread 0x017c1f thru 0x01d0d2 (0x54b4 bytes) 000545 1713430100.552117 write 0x03d65e thru 0x03ffff (0x29a2 bytes) 000546 1713430100.552735 write 0x0050c2 thru 0x014b04 (0xfa43 bytes) 000551 1713430100.557584 mapread 0x01d4f9 thru 0x026d89 (0x9891 bytes) 000552 1713430100.562892 read 0x003207 thru 0x009832 (0x662c bytes) 000555 1713430100.564067 mapread 0x013a95 thru 0x023420 (0xf98c bytes) 000556 1713430100.564745 write 0x038bb7 thru 0x03ffff (0x7449 bytes) 000557 1713430100.568359 write 0x03a69c thru 0x03ffff (0x5964 bytes) 000559 1713430100.569391 read 0x01bc4c thru 0x029182 (0xd537 bytes) 000565 1713430100.571188 trunc from 0x040000 to 0x009d7b (0x36284 bytes) 000567 1713430100.577633 write 0x03f3b1 thru 0x03ffff (0xc4f bytes) 000569 1713430100.578515 trunc from 0x040000 to 0x0092cc (0x36d33 bytes) 000572 1713430100.583944 read 0x0057d5 thru 0x0092cc (0x3af8 bytes) 000573 1713430100.584341 write 0x02ea9d thru 0x03d515 (0xea79 bytes) 000575 1713430100.585708 mapread 0x03a59e thru 0x03d515 (0x2f78 bytes) 000579 1713430100.586785 trunc from 0x03d516 to 0x01cb6d (0x209a8 bytes) 000581 1713430100.592793 write 0x0201e0 thru 0x02cbf0 (0xca11 bytes) 000584 1713430100.594170 mapread 0x005cad thru 0x0077d7 (0x1b2b bytes) 000585 1713430100.594709 write 0x00d50e thru 0x01c369 (0xee5c bytes) 000588 1713430100.601810 MIRROR_EXTEND to 3 mirrors 000590 1713430100.752540 mapwrite 0x011a98 thru 0x011c8f (0x1f8 bytes) 000593 1713430100.794017 read 0x001d76 thru 0x010c77 (0xef02 bytes) 000595 1713430100.797265 mapwrite 0x02ec0b thru 0x03bd09 (0xd0ff bytes) 000596 1713430100.830903 read 0x02696e thru 0x02a24b (0x38de bytes) 000597 1713430100.833979 write 0x0235b8 thru 0x0237f3 (0x23c bytes) 000598 1713430100.834589 trunc from 0x03bd0a to 0x030db6 (0xaf53 bytes) 000599 1713430100.839058 read 0x009fe6 thru 0x01318e (0x91a9 bytes) 000601 1713430100.839739 read 0x0125a8 thru 0x01ba18 (0x9471 bytes) 000604 1713430100.840778 mapwrite 0x02a44b thru 0x02d99a (0x3550 bytes) 000606 1713430100.857108 MIRROR_EXTEND to 4 mirrors 000608 1713430100.966901 mapread 0x000475 thru 0x006d63 (0x68ef bytes) 000609 1713430100.974116 write 0x03b86c thru 0x03ffff (0x4794 bytes) 000612 1713430100.988390 mapwrite 0x013aab thru 0x013f08 (0x45e bytes) 000613 1713430101.004899 MIRROR_RESYNC 4 mirrors 000615 1713430101.132246 mapread 0x02ddc3 thru 0x0353ce (0x760c bytes) 000617 1713430101.136054 write 0x02d419 thru 0x039c77 (0xc85f bytes) 000618 1713430101.153415 mapwrite 0x03a28d thru 0x03f4fd (0x5271 bytes) 000620 1713430101.173983 MIRROR_RESYNC 4 mirrors 000621 1713430101.299390 mapread 0x031c13 thru 0x03ffff (0xe3ed bytes) 000622 1713430101.305117 trunc from 0x040000 to 0x00725d (0x38da2 bytes) 000625 1713430101.324653 MIRROR_RESYNC 4 mirrors 000628 1713430101.450040 read 0x005a3a thru 0x00725d (0x1824 bytes) 000630 1713430101.451813 trunc from 0x00725e to 0x035a6a (0x2e80d bytes) 000631 1713430101.473304 mapwrite 0x02717e thru 0x02fffb (0x8e7e bytes) 000633 1713430101.501336 trunc from 0x035a6b to 0x028735 (0xd335 bytes) 000635 1713430101.506662 mapwrite 0x038e99 thru 0x03d1ce (0x4336 bytes) 000638 1713430101.526934 read 0x005acd thru 0x01055f (0xaa93 bytes) 000639 1713430101.530760 mapread 0x01ed20 thru 0x02abd3 (0xbeb4 bytes) 000640 1713430101.532525 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000643 1713430101.577378 trunc from 0x03d1cf to 0x010207 (0x2cfc7 bytes) 000644 1713430101.583216 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000645 1713430101.629197 MIRROR_RESYNC 2 mirrors 000646 1713430101.727242 mapread 0x00a917 thru 0x010207 (0x58f1 bytes) 000647 1713430101.732522 write 0x01ccab thru 0x020391 (0x36e7 bytes) 000648 1713430101.748864 trunc from 0x020392 to 0x0055b8 (0x1add9 bytes) 000649 1713430101.756065 mapread 0x002582 thru 0x0055b8 (0x3037 bytes) 000651 1713430101.760855 MIRROR_RESYNC 2 mirrors 000652 1713430101.879214 read 0x002262 thru 0x0055b8 (0x3357 bytes) 000655 1713430101.881066 trunc from 0x0055b9 to 0x02834a (0x22d92 bytes) 000656 1713430101.901336 write 0x03fed5 thru 0x03ffff (0x12b bytes) 000657 1713430101.903285 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000658 1713430101.940434 mapwrite 0x02c96f thru 0x0355a5 (0x8c37 bytes) 000660 1713430101.973090 mapwrite 0x01bb75 thru 0x02762c (0xbab8 bytes) 000661 1713430102.004016 write 0x0278c2 thru 0x032021 (0xa760 bytes) 000663 1713430102.005110 trunc from 0x040000 to 0x02703e (0x18fc1 bytes) 000664 1713430102.010444 mapwrite 0x03b081 thru 0x03ffff (0x4f7f bytes) 000665 1713430102.036817 MIRROR_EXTEND to 2 mirrors 000666 1713430102.177850 read 0x014d75 thru 0x01e207 (0x9493 bytes) 000669 1713430102.187931 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000670 1713430102.217168 write 0x03d04f thru 0x03ffff (0x2fb1 bytes) 000671 1713430102.221045 mapwrite 0x0080dd thru 0x014893 (0xc7b7 bytes) 000676 1713430102.248407 mapwrite 0x02fdfe thru 0x038b3d (0x8d40 bytes) 000682 1713430102.276329 read 0x006095 thru 0x00b874 (0x57e0 bytes) 000683 1713430102.276771 write 0x03963c thru 0x03b125 (0x1aea bytes) 000684 1713430102.277283 read 0x0106da thru 0x0136dd (0x3004 bytes) 000691 1713430102.278786 write 0x01b824 thru 0x024a21 (0x91fe bytes) 000696 1713430102.283832 mapread 0x03e1a8 thru 0x03ffff (0x1e58 bytes) 000699 1713430102.285226 mapwrite 0x02a120 thru 0x0385d2 (0xe4b3 bytes) 000700 1713430102.305519 trunc from 0x040000 to 0x014264 (0x2bd9b bytes) 000705 1713430102.312072 read 0x012890 thru 0x014264 (0x19d5 bytes) 000706 1713430102.312430 mapread 0x004922 thru 0x007287 (0x2966 bytes) 000710 1713430102.313686 write 0x0074f5 thru 0x00c89f (0x53ab bytes) 000711 1713430102.314181 mapwrite 0x010e9c thru 0x01382e (0x2993 bytes) 000712 1713430102.332209 mapread 0x007f79 thru 0x014264 (0xc2ec bytes) 000713 1713430102.332835 read 0x012b54 thru 0x014264 (0x1711 bytes) 000714 1713430102.333211 trunc from 0x014265 to 0x0119c9 (0x289b bytes) 000716 1713430102.337610 mapread 0x005e9b thru 0x00be9a (0x6000 bytes) 000719 1713430102.338769 mapwrite 0x00d601 thru 0x00d77f (0x17f bytes) 000723 1713430102.360851 mapread 0x00a1f4 thru 0x0119c9 (0x77d6 bytes) 000728 1713430102.364534 MIRROR_EXTEND to 2 mirrors 000731 1713430102.506224 mapwrite 0x01fc6a thru 0x029c29 (0x9fc0 bytes) 000732 1713430102.552988 read 0x012600 thru 0x01f20f (0xcc10 bytes) 000733 1713430102.557857 trunc from 0x029c2a to 0x00526e (0x249bb bytes) 000735 1713430102.563504 mapwrite 0x000cef thru 0x005afc (0x4e0e bytes) 000736 1713430102.586753 read 0x001051 thru 0x001622 (0x5d2 bytes) 000737 1713430102.587153 mapread 0x0015cd thru 0x005afc (0x4530 bytes) 000738 1713430102.587823 trunc from 0x005afd to 0x03471a (0x2ec1e bytes) 000739 1713430102.591650 write 0x0251fe thru 0x02b914 (0x6717 bytes) 000742 1713430102.596550 mapread 0x02f2b0 thru 0x03471a (0x546b bytes) 000743 1713430102.597090 read 0x02c3dc thru 0x03471a (0x833f bytes) 000744 1713430102.598800 MIRROR_EXTEND to 3 mirrors 000745 1713430102.733260 write 0x014c97 thru 0x016a65 (0x1dcf bytes) 000748 1713430102.750697 trunc from 0x03471b to 0x001c15 (0x32b05 bytes) 000750 1713430102.758931 mapread 0x000e8c thru 0x001c15 (0xd8a bytes) 000752 1713430102.762167 trunc from 0x001c16 to 0x02039a (0x1e785 bytes) 000753 1713430102.766037 trunc from 0x02039b to 0x021124 (0xd8a bytes) 000755 1713430102.770104 trunc from 0x021125 to 0x019efd (0x7227 bytes) 000756 1713430102.774089 trunc from 0x019efe to 0x03ada8 (0x20eab bytes) 000757 1713430102.777791 read 0x03146d thru 0x0364eb (0x507f bytes) 000759 1713430102.780499 mapread 0x033491 thru 0x03ada8 (0x7918 bytes) 000760 1713430102.782535 trunc from 0x03ada9 to 0x03772d (0x367b bytes) 000763 1713430102.786336 trunc from 0x03772e to 0x019d30 (0x1d9fd bytes) 000764 1713430102.789812 trunc from 0x019d31 to 0x02d309 (0x135d9 bytes) 000767 1713430102.794015 write 0x035a7d thru 0x03f965 (0x9ee9 bytes) 000768 1713430102.794752 mapread 0x00bc9b thru 0x010a93 (0x4df9 bytes) 000769 1713430102.797813 write 0x021e33 thru 0x02ff75 (0xe143 bytes) 000776 1713430102.803821 mapread 0x00971f thru 0x00bb11 (0x23f3 bytes) 000778 1713430102.804373 mapwrite 0x03a67b thru 0x03ffff (0x5985 bytes) truncating to largest ever: 0x3ff2d 000779 1713430102.823090 trunc from 0x040000 to 0x03ff2c (0xd3 bytes) 000780 1713430102.826357 write 0x011b8b thru 0x018fa4 (0x741a bytes) 000782 1713430102.828665 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000783 1713430102.871716 mapread 0x01d47c thru 0x026643 (0x91c8 bytes) 000784 1713430102.876683 read 0x011b72 thru 0x018003 (0x6492 bytes) 000785 1713430102.878857 mapwrite 0x000ddf thru 0x0055f6 (0x4818 bytes) 000786 1713430102.897779 read 0x03d14f thru 0x03ff2c (0x2dde bytes) 000787 1713430102.900261 mapread 0x00f20e thru 0x01b159 (0xbf4c bytes) 000788 1713430102.900698 read 0x000090 thru 0x00f399 (0xf30a bytes) 000789 1713430102.901016 mapwrite 0x010cb6 thru 0x014e8d (0x41d8 bytes) 000791 1713430102.914771 read 0x00aa34 thru 0x00c5bb (0x1b88 bytes) 000792 1713430102.915063 read 0x0204bf thru 0x027712 (0x7254 bytes) 000793 1713430102.915306 write 0x00d839 thru 0x00fcff (0x24c7 bytes) 000794 1713430102.917166 MIRROR_EXTEND to 3 mirrors 000796 1713430103.046298 write 0x00d0f7 thru 0x015735 (0x863f bytes) 000798 1713430103.065682 MIRROR_EXTEND to 4 mirrors 000799 1713430103.210535 trunc from 0x03ff2d to 0x024153 (0x1bdd9 bytes) 000800 1713430103.226275 read 0x00b8f5 thru 0x019457 (0xdb63 bytes) 000802 1713430103.229202 read 0x00a3cf thru 0x0195fa (0xf22c bytes) 000805 1713430103.232807 mapwrite 0x00a68c thru 0x01662b (0xbfa0 bytes) 000807 1713430103.250445 mapread 0x01e32e thru 0x01fa67 (0x173a bytes) 000810 1713430103.251381 write 0x0062b9 thru 0x01024c (0x9f94 bytes) 000811 1713430103.253502 MIRROR_RESYNC 4 mirrors 000813 1713430103.398651 trunc from 0x024154 to 0x025b93 (0x1a40 bytes) 000814 1713430103.419615 mapwrite 0x010709 thru 0x01a7ce (0xa0c6 bytes) 000816 1713430103.439008 read 0x00e3e8 thru 0x01bab0 (0xd6c9 bytes) 000817 1713430103.441971 MIRROR_EXTEND to 5 mirrors 000820 1713430103.551037 mapwrite 0x01fdac thru 0x022437 (0x268c bytes) 000822 1713430103.600366 read 0x0009c7 thru 0x001214 (0x84e bytes) 000823 1713430103.602445 trunc from 0x025b94 to 0x027af9 (0x1f66 bytes) 000824 1713430103.610706 trunc from 0x027afa to 0x01b224 (0xc8d5 bytes) 000827 1713430103.615366 trunc from 0x01b225 to 0x01e0f5 (0x2ed1 bytes) 000828 1713430103.619131 read 0x000187 thru 0x004cc2 (0x4b3c bytes) 000829 1713430103.622262 trunc from 0x01e0f6 to 0x01cc82 (0x1473 bytes) 000831 1713430103.627522 MIRROR_EXTEND to 6 mirrors 000833 1713430103.731750 read 0x000c9c thru 0x004e32 (0x4197 bytes) 000835 1713430103.736486 read 0x01a959 thru 0x01cc82 (0x232a bytes) 000837 1713430103.741761 mapread 0x00b241 thru 0x015f70 (0xad30 bytes) 000838 1713430103.744912 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000839 1713430103.787871 mapwrite 0x0385fe thru 0x038fe2 (0x9e5 bytes) 000840 1713430103.814583 trunc from 0x038fe3 to 0x0237ce (0x15814 bytes) 000842 1713430103.819924 trunc from 0x0237cf to 0x008da7 (0x1aa27 bytes) 000850 1713430103.826805 trunc from 0x008da8 to 0x00fcc5 (0x6f1e bytes) 000852 1713430103.832879 write 0x02ede7 thru 0x035284 (0x649e bytes) 000853 1713430103.833727 mapwrite 0x037292 thru 0x03f002 (0x7d71 bytes) 000854 1713430103.868259 mapread 0x039785 thru 0x03a30e (0xb8a bytes) 000855 1713430103.868837 write 0x01cda6 thru 0x029381 (0xc5dc bytes) 000856 1713430103.874437 write 0x03ea55 thru 0x03ffff (0x15ab bytes) 000857 1713430103.874997 read 0x01e327 thru 0x02c974 (0xe64e bytes) 000860 1713430103.877797 write 0x0382ea thru 0x03ffff (0x7d16 bytes) 000862 1713430103.878740 mapread 0x004eb9 thru 0x0135f5 (0xe73d bytes) 000863 1713430103.885166 mapread 0x020112 thru 0x02202a (0x1f19 bytes) 000864 1713430103.885694 read 0x03d741 thru 0x03ffff (0x28bf bytes) 000865 1713430103.887724 MIRROR_EXTEND to 6 mirrors 000870 1713430104.075434 write 0x0220a0 thru 0x0279ab (0x590c bytes) 000872 1713430104.095501 mapread 0x02880f thru 0x034328 (0xbb1a bytes) 000873 1713430104.101052 mapread 0x0058cf thru 0x006285 (0x9b7 bytes) 000874 1713430104.103934 mapread 0x02d5dd thru 0x031ca5 (0x46c9 bytes) 000875 1713430104.104438 write 0x02d293 thru 0x0392c3 (0xc031 bytes) 000876 1713430104.105148 write 0x01e361 thru 0x027979 (0x9619 bytes) 000877 1713430104.109830 MIRROR_EXTEND to 7 mirrors 000878 1713430104.253999 mapwrite 0x000776 thru 0x00ff7b (0xf806 bytes) 000879 1713430104.307590 read 0x035d0d thru 0x03d892 (0x7b86 bytes) 000883 1713430104.310700 read 0x001dcc thru 0x00cb2c (0xad61 bytes) 000888 1713430104.311909 mapread 0x01116f thru 0x0174d1 (0x6363 bytes) 000890 1713430104.316453 MIRROR_EXTEND to 8 mirrors 000893 1713430104.436402 read 0x03029e thru 0x031834 (0x1597 bytes) 000894 1713430104.438660 read 0x019bb5 thru 0x01e3ce (0x481a bytes) 000895 1713430104.442871 MIRROR_RESYNC 8 mirrors 000896 1713430104.665928 write 0x02a962 thru 0x02f455 (0x4af4 bytes) 000898 1713430104.683920 trunc from 0x040000 to 0x03ad65 (0x529a bytes) 000901 1713430104.692712 mapread 0x00d70d thru 0x00fd0a (0x25fe bytes) 000902 1713430104.695281 mapwrite 0x027ce1 thru 0x034056 (0xc376 bytes) 000904 1713430104.717138 read 0x014161 thru 0x019554 (0x53f4 bytes) 000905 1713430104.719453 read 0x00baf5 thru 0x019a37 (0xdf43 bytes) 000906 1713430104.719937 mapwrite 0x030102 thru 0x0367bc (0x66bb bytes) 000909 1713430104.742871 mapwrite 0x019188 thru 0x01a1ad (0x1026 bytes) 000910 1713430104.762325 read 0x02c7e6 thru 0x03509a (0x88b5 bytes) 000911 1713430104.762778 write 0x029a4e thru 0x02ca12 (0x2fc5 bytes) 000913 1713430104.763739 mapread 0x02917a thru 0x02d0c3 (0x3f4a bytes) 000914 1713430104.764287 read 0x0129ed thru 0x01d317 (0xa92b bytes) 000919 1713430104.765787 write 0x034fb9 thru 0x03fc41 (0xac89 bytes) 000921 1713430104.768609 MIRROR_EXTEND to 9 mirrors 000922 1713430104.903634 read 0x00ebd2 thru 0x010d61 (0x2190 bytes) 000923 1713430104.907349 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000924 1713430104.968434 mapread 0x0375e9 thru 0x03a88c (0x32a4 bytes) 000925 1713430104.978629 MIRROR_RESYNC 8 mirrors 000926 1713430105.219748 mapread 0x021b03 thru 0x02c772 (0xac70 bytes) 000930 1713430105.227139 mapwrite 0x00e8e8 thru 0x01e25e (0xf977 bytes) 000935 1713430105.283891 MIRROR_RESYNC 8 mirrors 000936 1713430105.536526 read 0x02ae6d thru 0x02b567 (0x6fb bytes) 000939 1713430105.538269 read 0x01db5c thru 0x01f7fa (0x1c9f bytes) 000940 1713430105.541477 read 0x027db4 thru 0x027ee0 (0x12d bytes) 000941 1713430105.541708 mapread 0x02c695 thru 0x02e5d5 (0x1f41 bytes) 000943 1713430105.543502 MIRROR_EXTEND to 9 mirrors 000945 1713430105.687281 mapwrite 0x03c796 thru 0x03ffff (0x386a bytes) 000947 1713430105.729932 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000950 1713430105.783117 mapwrite 0x02f5be thru 0x0353d3 (0x5e16 bytes) 000951 1713430105.803893 trunc from 0x040000 to 0x007a49 (0x385b6 bytes) 000952 1713430105.808708 mapread 0x0051b5 thru 0x007a49 (0x2895 bytes) 000954 1713430105.812301 write 0x00cbf6 thru 0x0167c3 (0x9bce bytes) 000955 1713430105.813260 mapread 0x0053a4 thru 0x010c33 (0xb890 bytes) 000956 1713430105.815394 trunc from 0x0167c4 to 0x00c6d8 (0xa0eb bytes) 000957 1713430105.819733 trunc from 0x00c6d9 to 0x00bc25 (0xab3 bytes) 000958 1713430105.823291 mapread 0x003f1c thru 0x003fe1 (0xc6 bytes) 000961 1713430105.825724 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000963 1713430105.872522 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000964 1713430105.913313 trunc from 0x00bc26 to 0x03ac17 (0x2eff2 bytes) 000966 1713430105.917058 write 0x02f49d thru 0x03ce50 (0xd9b4 bytes) 000968 1713430105.919294 trunc from 0x03ce51 to 0x0014c7 (0x3b989 bytes) 000969 1713430105.923471 mapread 0x000246 thru 0x0014c7 (0x1282 bytes) 000971 1713430105.927947 MIRROR_EXTEND to 7 mirrors 000972 1713430106.034249 mapread 0x000041 thru 0x0014c7 (0x1487 bytes) 000973 1713430106.038476 mapread 0x0002ea thru 0x0014c7 (0x11de bytes) 000975 1713430106.043999 write 0x01abd1 thru 0x022040 (0x7470 bytes) 000976 1713430106.058015 mapread 0x017487 thru 0x021a60 (0xa5da bytes) 000977 1713430106.062113 mapwrite 0x01f28f thru 0x025244 (0x5fb6 bytes) 000979 1713430106.083646 write 0x00768b thru 0x00e482 (0x6df8 bytes) 000982 1713430106.088819 MIRROR_EXTEND to 8 mirrors 000983 1713430106.200017 MIRROR_EXTEND to 9 mirrors 000989 1713430106.310476 write 0x0095f8 thru 0x01714f (0xdb58 bytes) 000992 1713430106.326968 write 0x0281eb thru 0x03331b (0xb131 bytes) 000993 1713430106.327696 mapread 0x02a8f4 thru 0x02aa07 (0x114 bytes) 000995 1713430106.329450 MIRROR_EXTEND to 10 mirrors All operations completed A-OK! PASS 70a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 04:48:29 (1713430109) Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7223552 .. Extending file size to 2371552 .. done Extending file size to 4803072 .. Extending file size to 6201568 .. Extending file size to 8419904 .. Extending file size to 8172192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8507456 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9300832 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8508896 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8535488 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9193088 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 6668 6669 6670 6672 6675 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:49:33 (1713430173) Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3971680 .. Extending file size to 8448096 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6435616 .. Extending file size to 7978048 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8642176 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8805056 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9273824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8693344 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8906048 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 10240 10241 10243 10244 10246 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:50:38 (1713430238) Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0281309 s, 74.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0693146 s, 30.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:50:44 (1713430244) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.030437 s, 68.9 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 04:50:47 (1713430247) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc55:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc55:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc56:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xc58:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:50:50 (1713430250) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0302313 s, 69.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc59:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:50:54 (1713430254) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0300705 s, 69.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc58:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc59:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc58:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc59:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5c:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:50:57 (1713430257) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0237534 s, 88.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5e:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:51:21 (1713430281) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180788 s, 58.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc60:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc60:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5c:0x0] } PASS 204d (42s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:52:05 (1713430325) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.024847 s, 84.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc61:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc61:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (42s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:52:49 (1713430369) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. Creating new pool oleg127-server: Pool lustre.test_204f created Adding targets to pool oleg127-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg127-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.021836 s, 96.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc62:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc62:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg127-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg127-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg127-server: Pool lustre.test_204f destroyed PASS 204f (56s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 04:53:46 (1713430426) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:53:50 (1713430430) oleg127-server: Pool lustre.test_206 created Adding targets to pool oleg127-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg127-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg127-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg127-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg127-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:54:03 (1713430443) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143496 s, 73.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00716294 s, 146 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x5ab2ce99 Mirror 3: 0x5ab2ce99 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:54:07 (1713430447) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:54:09 (1713430449) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:54:11 (1713430451) Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0143434 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2440000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d80000 Read this (4 bytes): BBBB 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:54:26 (1713430466) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00346964 s, 2.4 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 04:54:30 (1713430470) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113619 s, 92.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69c lmm_fid: [0x200000404:0x69c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69c lmm_fid: [0x200000404:0x69c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc68:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835612 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-18 04:54:30.000000000 -0400 Modify: 2024-04-18 04:54:30.000000000 -0400 Change: 2024-04-18 04:54:30.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:54:33 (1713430473) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 866 sec ============ 04:54:35 (1713430475) === sanity-flr: start cleanup 04:54:35 (1713430475) === === sanity-flr: finish cleanup 04:54:36 (1713430476) ===