-----============= acceptance-small: sanity-flr ============----- Tue Apr 16 10:49:34 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:49:37 (1713278977) === oleg203-client.virtnet: executing check_config_client /mnt/lustre oleg203-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg203-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012aab8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aab8800.idle_timeout=debug disable quota as required oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:49:43 (1713278983) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6915] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 10:49:44 (1713278984) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:49:49 (1713278989) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:49:51 (1713278991) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 10:49:53 (1713278993) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:49:59 (1713278999) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:50:01 (1713279001) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 10:50:03 (1713279003) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012aab8800.stats=clear osc.lustre-OST0001-osc-ffff88012aab8800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:50:06 (1713279006) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:50:10 (1713279010) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:50:13 (1713279013) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 10:50:16 (1713279016) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 10:50:19 (1713279019) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:50:22 (1713279022) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:50:24 (1713279024) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:50:27 (1713279027) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 10:50:31 (1713279031) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:50:34 (1713279034) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.359541 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0450668 s, 23.3 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:50:38 (1713279038) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0315222 s, 166 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:50:41 (1713279041) Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aab8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aab8800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aab8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aab8800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:51:03 (1713279063) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0665354 s, 63.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aab8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aab8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:51:12 (1713279072) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aab8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aab8800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aab8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aab8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:51:27 (1713279087) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.73845 s, 71.0 MB/s -rw-r--r-- 1 root root 52428800 Apr 16 10:51 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.178s user 0m0.001s sys 0m0.147s Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aab8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aab8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.162s user 0m0.001s sys 0m0.097s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aab8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aab8800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.178s user 0m0.000s sys 0m0.155s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012aab8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aab8800.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:51:49 (1713279109) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:51:51 (1713279111) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:51:53 (1713279113) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 10:51:55 (1713279115) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 10:51:59 (1713279119) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.111912 s, 112 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:52:03 (1713279123) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00823462 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105575 s, 99.3 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:52:06 (1713279126) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:52:10 (1713279130) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000573523 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000595013 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000634076 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000438104 s, 2.3 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 10:52:14 (1713279134) create test files with size 15739800 .. 7e9960e81b99f4280a4a63684f07ed2e - 8dfec8844a276ab385654c48b9ac5091 - 558039203c917ef5a445f9eaa0304e04 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre PASS 37 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 10:52:20 (1713279140) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:52:24 (1713279144) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:52:27 (1713279147) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214537 s, 48.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x45:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00723052 s, 145 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x46:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 10:52:31 (1713279151) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0221836 s, 94.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0858707 s, 48.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0370062 s, 56.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 10:52:35 (1713279155) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 10:52:37 (1713279157) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:52:39 (1713279159) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00770687 s, 136 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 10:52:43 (1713279163) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:52:45 (1713279165) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:52:48 (1713279168) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0613971 s, 171 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:52:52 (1713279172) llite.lustre-ffff88012dae3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0699745 s, 150 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a9330000.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:52:57 (1713279177) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 10:53:04 (1713279184) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 10:53:11 (1713279191) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 10:53:13 (1713279193) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0589055 s, 53.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0467247 s, 67.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: ba9a0cb23796c31be85f356e9006ff85 - mirror 1 checksum: cf61de9268dd6f3a797f805c8f3e9196 - mirror 2 checksum: ba9a0cb23796c31be85f356e9006ff85 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 10:53:14 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 10:53:14 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ba9a0cb23796c31be85f356e9006ff85 - mirror 1 checksum: cf61de9268dd6f3a797f805c8f3e9196 - mirror 2 checksum: cf61de9268dd6f3a797f805c8f3e9196 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 10:53:14 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 10:53:14 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:53:17 (1713279197) Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 10:53:19 (1713279199) shut down Failover mds1 to oleg203-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 10:53:31 (1713279211) targets are mounted 10:53:31 (1713279211) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00800521 s, 131 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 50A (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:53:43 (1713279223) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 10:53:46 (1713279226) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:53:48 (1713279228) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00233545 s, 7.0 MB/s PASS 50c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 10:53:51 (1713279231) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:53:53 (1713279233) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:53:56 (1713279236) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:54:01 (1713279241) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:54:20 (1713279260) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:54:39 (1713279279) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:54:57 (1713279297) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 10:55:00 (1713279300) mirror create pid 26588 mirror split pid 26591 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26588 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26591 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 10:56:03 (1713279363) Seed set to 9363 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x973c 000001 1713279363.907014 trunc from 00000000 to 0x00973b (0x973c bytes) 000002 1713279363.910759 write 0x02ac92 thru 0x02d301 (0x2670 bytes) 000003 1713279363.911605 write 0x034783 thru 0x03a081 (0x58ff bytes) 000005 1713279363.913847 mapread 0x027495 thru 0x029573 (0x20df bytes) 000006 1713279363.918026 write 0x00d0e3 thru 0x017673 (0xa591 bytes) 000010 1713279363.922061 mapread 0x01df4b thru 0x01e328 (0x3de bytes) truncating to largest ever: 0x3c941 000014 1713279363.924674 trunc from 0x03a082 to 0x03c940 (0x28bf bytes) 000015 1713279363.928119 mapwrite 0x010877 thru 0x013dc5 (0x354f bytes) 000017 1713279363.951621 trunc from 0x03c941 to 0x03c471 (0x4cf bytes) 000018 1713279363.955845 trunc from 0x03c472 to 0x035281 (0x71f0 bytes) 000021 1713279363.959654 mapread 0x0171c2 thru 0x0215fb (0xa43a bytes) 000023 1713279363.960336 write 0x00f276 thru 0x016406 (0x7191 bytes) 000025 1713279363.961480 mapread 0x021c5c thru 0x024800 (0x2ba5 bytes) 000027 1713279363.962182 mapread 0x01bca9 thru 0x01c017 (0x36f bytes) 000028 1713279363.962577 mapwrite 0x008657 thru 0x008ef9 (0x8a3 bytes) 000029 1713279363.977072 trunc from 0x035282 to 0x0285a0 (0xcce1 bytes) 000030 1713279363.980973 mapread 0x023f2f thru 0x0285a0 (0x4672 bytes) 000032 1713279363.981814 mapwrite 0x00a83b thru 0x00b0ad (0x873 bytes) truncating to largest ever: 0x3eeb2 000034 1713279363.994852 trunc from 0x0285a1 to 0x03eeb1 (0x16911 bytes) 000036 1713279363.998594 write 0x039b60 thru 0x03ffff (0x64a0 bytes) 000038 1713279364.001801 mapwrite 0x0232f5 thru 0x026ee7 (0x3bf3 bytes) 000039 1713279364.017377 write 0x021c19 thru 0x02a4bc (0x88a4 bytes) 000040 1713279364.020946 MIRROR_EXTEND to 2 mirrors 000042 1713279364.147578 mapwrite 0x02ec35 thru 0x037724 (0x8af0 bytes) 000045 1713279364.196234 mapwrite 0x00f035 thru 0x017846 (0x8812 bytes) 000046 1713279364.213447 MIRROR_EXTEND to 3 mirrors 000047 1713279364.328527 mapwrite 0x01926d thru 0x01ecdb (0x5a6f bytes) 000048 1713279364.373319 trunc from 0x040000 to 0x004d5e (0x3b2a1 bytes) 000049 1713279364.382049 write 0x01245a thru 0x01ac04 (0x87ab bytes) 000050 1713279364.383211 write 0x029833 thru 0x0352f2 (0xbac0 bytes) 000051 1713279364.386050 MIRROR_EXTEND to 4 mirrors 000052 1713279364.486407 mapwrite 0x022aed thru 0x032936 (0xfe4a bytes) 000058 1713279364.533510 read 0x01e841 thru 0x01ebb3 (0x373 bytes) 000059 1713279364.533851 mapread 0x00c023 thru 0x00e48c (0x246a bytes) 000060 1713279364.536441 read 0x01300f thru 0x01932e (0x6320 bytes) 000061 1713279364.538802 read 0x016ee4 thru 0x01e8c7 (0x79e4 bytes) 000062 1713279364.539113 trunc from 0x0352f3 to 0x00436f (0x30f83 bytes) 000063 1713279364.546866 read 0x00386e thru 0x00436f (0xb02 bytes) 000065 1713279364.547408 write 0x017b88 thru 0x01fe92 (0x830b bytes) 000067 1713279364.548278 read 0x01d48b thru 0x01fd11 (0x2887 bytes) 000068 1713279364.548562 mapread 0x00b73c thru 0x0122cc (0x6b91 bytes) 000069 1713279364.550834 write 0x017018 thru 0x01b75c (0x4745 bytes) 000070 1713279364.551322 trunc from 0x01fe93 to 0x03ec0b (0x1ed79 bytes) 000072 1713279364.555102 read 0x02dae5 thru 0x0340c3 (0x65df bytes) 000073 1713279364.557979 trunc from 0x03ec0c to 0x03828b (0x6980 bytes) 000074 1713279364.562475 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000075 1713279364.613676 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000076 1713279364.655174 write 0x01349f thru 0x021b4c (0xe6ae bytes) 000081 1713279364.659980 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000084 1713279364.689527 write 0x00f7f1 thru 0x00fa7b (0x28b bytes) 000086 1713279364.691364 read 0x006085 thru 0x00e19c (0x8118 bytes) 000087 1713279364.693634 read 0x00e999 thru 0x012c20 (0x4288 bytes) 000089 1713279364.697047 write 0x03f811 thru 0x03ffff (0x7ef bytes) 000097 1713279364.699279 trunc from 0x040000 to 0x0099aa (0x36655 bytes) 000098 1713279364.703458 write 0x02ed20 thru 0x03d46b (0xe74c bytes) 000099 1713279364.704703 mapread 0x000eaa thru 0x00e5ef (0xd746 bytes) 000100 1713279364.710625 mapwrite 0x014ef4 thru 0x01c3ab (0x74b8 bytes) 000102 1713279364.736764 mapread 0x012863 thru 0x0129fc (0x19a bytes) 000107 1713279364.740129 read 0x024fad thru 0x033a00 (0xea54 bytes) 000109 1713279364.743361 write 0x00a912 thru 0x019e0e (0xf4fd bytes) 000111 1713279364.744473 mapread 0x021bfe thru 0x022d50 (0x1153 bytes) 000112 1713279364.745105 mapwrite 0x00637e thru 0x00d193 (0x6e16 bytes) 000113 1713279364.760794 mapread 0x03402b thru 0x037c0e (0x3be4 bytes) 000121 1713279364.762871 read 0x02ad53 thru 0x02d38f (0x263d bytes) 000123 1713279364.763507 read 0x02b069 thru 0x035e03 (0xad9b bytes) 000124 1713279364.765669 MIRROR_EXTEND to 2 mirrors 000125 1713279364.865539 mapread 0x028118 thru 0x033937 (0xb820 bytes) 000126 1713279364.872732 mapread 0x00bf21 thru 0x00cd86 (0xe66 bytes) 000127 1713279364.876364 trunc from 0x03d46c to 0x00af78 (0x324f3 bytes) 000128 1713279364.898430 trunc from 0x00af79 to 0x017ae5 (0xcb6d bytes) 000133 1713279364.904292 read 0x002a59 thru 0x00c770 (0x9d18 bytes) 000138 1713279364.908360 write 0x01d90b thru 0x0206c5 (0x2dbb bytes) 000139 1713279364.909170 read 0x0052af thru 0x011b01 (0xc853 bytes) 000140 1713279364.911896 mapwrite 0x037a99 thru 0x03ffff (0x8567 bytes) 000142 1713279364.934168 read 0x015f8b thru 0x022cdc (0xcd52 bytes) 000144 1713279364.936124 MIRROR_RESYNC 2 mirrors 000146 1713279365.029379 write 0x02de30 thru 0x037b63 (0x9d34 bytes) 000147 1713279365.043554 write 0x00fe10 thru 0x013777 (0x3968 bytes) 000150 1713279365.048283 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000151 1713279365.077861 mapread 0x03373c thru 0x03ffff (0xc8c4 bytes) 000152 1713279365.083379 trunc from 0x040000 to 0x007d5c (0x382a3 bytes) 000153 1713279365.089744 read 0x0064a3 thru 0x007d5c (0x18ba bytes) 000155 1713279365.091470 write 0x012b0a thru 0x0131cf (0x6c6 bytes) 000159 1713279365.092784 write 0x03fae5 thru 0x03ffff (0x51b bytes) 000160 1713279365.093264 read 0x03b9a6 thru 0x03ffff (0x465a bytes) 000161 1713279365.096205 mapread 0x012faa thru 0x01f0b5 (0xc10c bytes) 000162 1713279365.096671 read 0x01563c thru 0x0200d5 (0xaa9a bytes) 000163 1713279365.096964 write 0x00b599 thru 0x013d59 (0x87c1 bytes) 000165 1713279365.097648 write 0x00a2cf thru 0x0184be (0xe1f0 bytes) 000166 1713279365.098297 mapread 0x030764 thru 0x03d0a1 (0xc93e bytes) 000167 1713279365.098693 mapread 0x0327e6 thru 0x0367c0 (0x3fdb bytes) 000168 1713279365.099111 mapread 0x02b2b5 thru 0x036863 (0xb5af bytes) 000174 1713279365.102034 MIRROR_EXTEND to 2 mirrors 000176 1713279365.213453 trunc from 0x040000 to 0x003ca1 (0x3c35e bytes) 000177 1713279365.235520 MIRROR_EXTEND to 3 mirrors 000179 1713279365.326988 read 0x001c38 thru 0x003ca1 (0x206a bytes) 000181 1713279365.330150 trunc from 0x003ca2 to 0x02ee86 (0x2b1e5 bytes) 000182 1713279365.349406 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000183 1713279365.392765 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000184 1713279365.418195 mapread 0x00f590 thru 0x014440 (0x4eb1 bytes) 000185 1713279365.423147 write 0x018c5b thru 0x01e4d7 (0x587d bytes) 000190 1713279365.425439 mapwrite 0x03077c thru 0x03fb79 (0xf3fe bytes) 000191 1713279365.449458 write 0x0143c2 thru 0x021355 (0xcf94 bytes) 000192 1713279365.451379 write 0x009213 thru 0x0187eb (0xf5d9 bytes) 000193 1713279365.453960 read 0x03005a thru 0x032573 (0x251a bytes) 000194 1713279365.454338 trunc from 0x03fb7a to 0x01a58d (0x255ec bytes) 000195 1713279365.457947 mapwrite 0x0259f7 thru 0x02e8d4 (0x8ede bytes) 000196 1713279365.479652 mapwrite 0x0348de thru 0x035ff6 (0x1719 bytes) 000199 1713279365.497866 read 0x00c102 thru 0x0171d0 (0xb0cf bytes) 000200 1713279365.498299 trunc from 0x035ff7 to 0x001cc3 (0x34333 bytes) 000204 1713279365.503380 mapread 0x0000fc thru 0x001cc3 (0x1bc8 bytes) 000205 1713279365.505161 mapread 0x000958 thru 0x001cc3 (0x136c bytes) 000206 1713279365.505565 mapwrite 0x02eb9d thru 0x0354e9 (0x694d bytes) 000212 1713279365.529731 mapwrite 0x030e4b thru 0x036ab8 (0x5c6e bytes) 000213 1713279365.549384 read 0x033c18 thru 0x036ab8 (0x2ea1 bytes) 000214 1713279365.549829 trunc from 0x036ab9 to 0x005b94 (0x30f24 bytes) 000215 1713279365.553400 read 0x0017e3 thru 0x005b94 (0x43b2 bytes) 000216 1713279365.553792 mapread 0x00100f thru 0x005b94 (0x4b86 bytes) 000217 1713279365.555775 MIRROR_EXTEND to 2 mirrors 000222 1713279365.659855 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000223 1713279365.686385 read 0x00584e thru 0x005b94 (0x347 bytes) 000224 1713279365.688221 trunc from 0x005b95 to 0x02714c (0x215b8 bytes) 000225 1713279365.697334 mapwrite 0x021fcc thru 0x029339 (0x736e bytes) 000227 1713279365.720375 read 0x002e95 thru 0x0086cf (0x583b bytes) 000228 1713279365.723337 read 0x0017e4 thru 0x010692 (0xeeaf bytes) 000229 1713279365.723657 mapread 0x00a1a5 thru 0x0183c1 (0xe21d bytes) 000230 1713279365.724116 mapwrite 0x03a32e thru 0x03ffff (0x5cd2 bytes) 000231 1713279365.748694 mapread 0x03dbbf thru 0x03ffff (0x2441 bytes) 000233 1713279365.749737 read 0x02fb9e thru 0x03d216 (0xd679 bytes) 000234 1713279365.750232 mapwrite 0x013d1a thru 0x022ebe (0xf1a5 bytes) 000235 1713279365.766590 trunc from 0x040000 to 0x00a35b (0x35ca4 bytes) 000236 1713279365.771207 mapwrite 0x00ccce thru 0x012b23 (0x5e56 bytes) 000237 1713279365.791159 mapread 0x00c6b5 thru 0x012b23 (0x646f bytes) 000239 1713279365.791782 read 0x0032bf thru 0x0083ba (0x50fc bytes) 000240 1713279365.792192 mapread 0x00b3b0 thru 0x012b23 (0x7774 bytes) 000241 1713279365.792751 mapwrite 0x01f404 thru 0x020827 (0x1424 bytes) 000244 1713279365.812316 mapread 0x004b57 thru 0x010b19 (0xbfc3 bytes) 000245 1713279365.813075 mapwrite 0x021e1b thru 0x024c6d (0x2e53 bytes) 000246 1713279365.830705 read 0x00bdef thru 0x0127a4 (0x69b6 bytes) 000248 1713279365.831436 mapread 0x003ef1 thru 0x00d7d0 (0x98e0 bytes) 000250 1713279365.832323 mapwrite 0x0237b6 thru 0x02ea25 (0xb270 bytes) 000252 1713279365.854177 MIRROR_EXTEND to 2 mirrors 000253 1713279365.955869 mapwrite 0x02a9b7 thru 0x031253 (0x689d bytes) 000255 1713279365.995440 mapwrite 0x00d0c9 thru 0x012798 (0x56d0 bytes) 000256 1713279366.010081 trunc from 0x031254 to 0x0361bb (0x4f68 bytes) 000259 1713279366.015269 mapwrite 0x031736 thru 0x03cef8 (0xb7c3 bytes) 000260 1713279366.036150 read 0x031c3d thru 0x03a5bf (0x8983 bytes) 000263 1713279366.037931 MIRROR_RESYNC 2 mirrors 000265 1713279366.125238 mapwrite 0x00b8b3 thru 0x010935 (0x5083 bytes) 000267 1713279366.169490 trunc from 0x03cef9 to 0x00614d (0x36dab bytes) 000268 1713279366.177888 write 0x038751 thru 0x0388c7 (0x177 bytes) 000269 1713279366.178708 mapread 0x026fd0 thru 0x02c2e7 (0x5318 bytes) 000271 1713279366.181569 write 0x01c6d9 thru 0x028367 (0xbc8f bytes) 000272 1713279366.184055 trunc from 0x0388c8 to 0x031a88 (0x6e3f bytes) 000273 1713279366.187782 trunc from 0x031a89 to 0x01613f (0x1b949 bytes) 000277 1713279366.193153 MIRROR_RESYNC 2 mirrors 000278 1713279366.279155 trunc from 0x016140 to 0x034412 (0x1e2d3 bytes) 000279 1713279366.297567 mapwrite 0x0210c0 thru 0x024d14 (0x3c55 bytes) 000281 1713279366.316370 write 0x0040d8 thru 0x00b573 (0x749c bytes) 000284 1713279366.320002 trunc from 0x034413 to 0x0175c4 (0x1ce4e bytes) 000285 1713279366.324684 read 0x0167e5 thru 0x017551 (0xd6d bytes) 000286 1713279366.327363 MIRROR_EXTEND to 3 mirrors 000289 1713279366.430936 mapwrite 0x01d1e6 thru 0x02c6dd (0xf4f8 bytes) 000290 1713279366.471205 mapread 0x025eac thru 0x02b999 (0x5aee bytes) 000291 1713279366.471790 write 0x033892 thru 0x037e35 (0x45a4 bytes) 000292 1713279366.473729 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000293 1713279366.521577 MIRROR_RESYNC 2 mirrors 000294 1713279366.611767 read 0x001265 thru 0x00b927 (0xa6c3 bytes) 000295 1713279366.614110 read 0x015d44 thru 0x01a46b (0x4728 bytes) 000296 1713279366.617424 mapwrite 0x01bb03 thru 0x025c16 (0xa114 bytes) truncating to largest ever: 0x3f87d 000297 1713279366.669623 trunc from 0x037e36 to 0x03f87c (0x7a47 bytes) 000298 1713279366.678258 trunc from 0x03f87d to 0x03e81d (0x105f bytes) 000300 1713279366.683772 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000301 1713279366.714242 read 0x020c00 thru 0x02d5f6 (0xc9f7 bytes) 000302 1713279366.716920 mapwrite 0x01d0a2 thru 0x0295d4 (0xc533 bytes) 000307 1713279366.738143 mapread 0x006c8b thru 0x00faa4 (0x8e1a bytes) 000308 1713279366.742875 trunc from 0x03e81e to 0x001006 (0x3d817 bytes) 000309 1713279366.746773 mapread 0x00060c thru 0x001006 (0x9fb bytes) 000310 1713279366.748364 read 0x0009fd thru 0x001006 (0x60a bytes) 000311 1713279366.748651 trunc from 0x001007 to 0x0018b0 (0x8aa bytes) 000314 1713279366.752568 mapread 0x0008b7 thru 0x000d5a (0x4a4 bytes) 000318 1713279366.753953 read 0x00186f thru 0x0018b0 (0x42 bytes) 000320 1713279366.754668 read 0x0011db thru 0x0018b0 (0x6d6 bytes) 000322 1713279366.755520 mapread 0x000a53 thru 0x0018b0 (0xe5e bytes) 000323 1713279366.755902 write 0x02b6d9 thru 0x03976c (0xe094 bytes) 000328 1713279366.757786 mapread 0x027a40 thru 0x032c2a (0xb1eb bytes) 000329 1713279366.760710 mapwrite 0x00c755 thru 0x0128a7 (0x6153 bytes) 000330 1713279366.775527 trunc from 0x03976d to 0x016f0d (0x2285f bytes) 000332 1713279366.780975 MIRROR_EXTEND to 2 mirrors 000333 1713279366.877497 write 0x01a652 thru 0x02a58e (0xff3d bytes) 000335 1713279366.892213 mapwrite 0x017e9d thru 0x01fba7 (0x7d0b bytes) 000336 1713279366.912174 mapread 0x016398 thru 0x0168f3 (0x55c bytes) 000338 1713279366.912993 trunc from 0x02a58f to 0x008850 (0x21d3e bytes) 000339 1713279366.922431 write 0x0163ae thru 0x0163ff (0x52 bytes) 000340 1713279366.923111 mapwrite 0x002a74 thru 0x00c190 (0x971d bytes) 000342 1713279366.942702 read 0x00c92c thru 0x0163ff (0x9ad4 bytes) 000343 1713279366.944628 trunc from 0x016400 to 0x022f81 (0xcb82 bytes) 000344 1713279366.948177 mapread 0x003e17 thru 0x00fb5a (0xbd44 bytes) 000346 1713279366.948860 write 0x03cea9 thru 0x03ffff (0x3157 bytes) 000348 1713279366.949638 write 0x009843 thru 0x01404d (0xa80b bytes) 000349 1713279366.951236 MIRROR_RESYNC 2 mirrors 000350 1713279367.054229 read 0x01c6f7 thru 0x01cdf6 (0x700 bytes) 000351 1713279367.057048 mapwrite 0x004bca thru 0x012ee2 (0xe319 bytes) 000352 1713279367.109979 mapwrite 0x01bcf5 thru 0x01e2df (0x25eb bytes) 000353 1713279367.125937 mapwrite 0x0379ab thru 0x03d686 (0x5cdc bytes) 000354 1713279367.143289 mapread 0x012869 thru 0x0216ce (0xee66 bytes) 000357 1713279367.144851 write 0x00476f thru 0x00b83d (0x70cf bytes) 000360 1713279367.146182 write 0x0310fe thru 0x0319a0 (0x8a3 bytes) 000363 1713279367.147550 mapwrite 0x027d2b thru 0x029985 (0x1c5b bytes) 000364 1713279367.165755 mapwrite 0x002e8b thru 0x00a401 (0x7577 bytes) 000367 1713279367.180905 write 0x005f63 thru 0x008303 (0x23a1 bytes) 000368 1713279367.181366 write 0x00ec18 thru 0x0178c2 (0x8cab bytes) 000369 1713279367.182025 mapread 0x01572c thru 0x016880 (0x1155 bytes) 000370 1713279367.182416 mapwrite 0x01eb3a thru 0x025989 (0x6e50 bytes) 000372 1713279367.197912 mapread 0x0387a2 thru 0x03ffff (0x785e bytes) 000373 1713279367.199897 MIRROR_RESYNC 2 mirrors 000374 1713279367.289057 read 0x006b84 thru 0x00a658 (0x3ad5 bytes) 000376 1713279367.293010 mapread 0x02dfac thru 0x038e9a (0xaeef bytes) 000377 1713279367.300615 trunc from 0x040000 to 0x0069c3 (0x3963c bytes) 000378 1713279367.319192 write 0x034c58 thru 0x03ffff (0xb3a8 bytes) 000381 1713279367.322085 MIRROR_RESYNC 2 mirrors 000382 1713279367.420078 mapwrite 0x034763 thru 0x03ffff (0xb89d bytes) 000384 1713279367.468269 mapread 0x002216 thru 0x00ce42 (0xac2d bytes) 000387 1713279367.473644 trunc from 0x040000 to 0x00a966 (0x35699 bytes) 000388 1713279367.482470 mapwrite 0x023c1e thru 0x033a50 (0xfe33 bytes) 000389 1713279367.507284 mapwrite 0x0308f9 thru 0x039c7d (0x9385 bytes) 000390 1713279367.528860 read 0x0399ab thru 0x039c7d (0x2d3 bytes) 000394 1713279367.530317 trunc from 0x039c7e to 0x0101cb (0x29ab2 bytes) 000395 1713279367.535348 MIRROR_RESYNC 2 mirrors 000396 1713279367.619427 read 0x005a9b thru 0x00bb07 (0x606d bytes) 000399 1713279367.621495 read 0x007502 thru 0x00e723 (0x7222 bytes) 000401 1713279367.623640 read 0x0036f1 thru 0x00e3ab (0xacbb bytes) 000402 1713279367.623848 write 0x013b2e thru 0x014508 (0x9db bytes) 000403 1713279367.637085 trunc from 0x014509 to 0x02f72e (0x1b226 bytes) 000404 1713279367.644817 write 0x02fcd5 thru 0x033339 (0x3665 bytes) 000405 1713279367.647977 read 0x0059ce thru 0x00c00c (0x663f bytes) 000406 1713279367.648436 mapwrite 0x01954d thru 0x024586 (0xb03a bytes) 000407 1713279367.665454 MIRROR_RESYNC 2 mirrors 000408 1713279367.749963 write 0x00ea0e thru 0x0189c4 (0x9fb7 bytes) 000410 1713279367.766978 trunc from 0x03333a to 0x0327d8 (0xb61 bytes) 000411 1713279367.775266 read 0x02ed27 thru 0x0327d8 (0x3ab2 bytes) 000413 1713279367.777945 trunc from 0x0327d9 to 0x02ae0e (0x79ca bytes) 000414 1713279367.781340 write 0x007e6d thru 0x00f052 (0x71e6 bytes) 000415 1713279367.783747 mapread 0x022ec3 thru 0x026295 (0x33d3 bytes) 000418 1713279367.786431 mapwrite 0x018fa6 thru 0x02845f (0xf4ba bytes) 000419 1713279367.803794 trunc from 0x02ae0f to 0x007634 (0x237da bytes) 000420 1713279367.807806 write 0x02c6cd thru 0x034b16 (0x844a bytes) 000421 1713279367.808462 write 0x0191f2 thru 0x023a18 (0xa827 bytes) 000424 1713279367.813703 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000425 1713279367.843857 write 0x00f75e thru 0x01b930 (0xc1d3 bytes) 000426 1713279367.846479 mapread 0x022d11 thru 0x026e63 (0x4153 bytes) 000427 1713279367.848868 read 0x00a500 thru 0x01606c (0xbb6d bytes) 000428 1713279367.851106 trunc from 0x034b17 to 0x01d7f1 (0x17325 bytes) 000429 1713279367.854801 read 0x001f21 thru 0x005b07 (0x3be7 bytes) 000431 1713279367.857971 mapread 0x011da8 thru 0x016090 (0x42e9 bytes) 000435 1713279367.859422 write 0x0207a2 thru 0x023043 (0x28a2 bytes) 000436 1713279367.860163 mapwrite 0x01a832 thru 0x025d19 (0xb4e8 bytes) 000437 1713279367.881864 read 0x01a637 thru 0x01e644 (0x400e bytes) 000439 1713279367.882743 write 0x000a66 thru 0x005b84 (0x511f bytes) 000440 1713279367.883573 trunc from 0x025d1a to 0x022364 (0x39b5 bytes) 000442 1713279367.887984 trunc from 0x022365 to 0x013d10 (0xe654 bytes) 000445 1713279367.892477 trunc from 0x013d11 to 0x026bd7 (0x12ec7 bytes) 000446 1713279367.895980 write 0x02b6fe thru 0x038a20 (0xd323 bytes) 000453 1713279367.898127 trunc from 0x038a21 to 0x013417 (0x25609 bytes) 000455 1713279367.901579 mapread 0x002657 thru 0x00a293 (0x7c3d bytes) 000456 1713279367.902138 mapread 0x006132 thru 0x00bc03 (0x5ad2 bytes) 000459 1713279367.903184 trunc from 0x013418 to 0x0182e8 (0x4ed1 bytes) 000460 1713279367.906332 write 0x00a23b thru 0x00ee99 (0x4c5f bytes) 000461 1713279367.906978 mapread 0x00dc55 thru 0x0182e8 (0xa694 bytes) 000463 1713279367.909390 write 0x02c52d thru 0x0351d2 (0x8ca6 bytes) 000464 1713279367.910125 write 0x00dcb5 thru 0x00dd11 (0x5d bytes) 000466 1713279367.910577 write 0x01dc4b thru 0x025bcf (0x7f85 bytes) 000467 1713279367.912795 write 0x00165d thru 0x00c481 (0xae25 bytes) 000468 1713279367.913446 mapwrite 0x01d053 thru 0x024698 (0x7646 bytes) 000469 1713279367.928594 mapwrite 0x014d34 thru 0x01eeac (0xa179 bytes) 000470 1713279367.942503 mapread 0x02ea8b thru 0x034877 (0x5ded bytes) 000471 1713279367.943254 write 0x03a700 thru 0x03a9e0 (0x2e1 bytes) 000472 1713279367.944020 mapread 0x02d046 thru 0x032448 (0x5403 bytes) 000474 1713279367.945063 write 0x019041 thru 0x01e3ca (0x538a bytes) 000476 1713279367.946244 mapwrite 0x030800 thru 0x03450d (0x3d0e bytes) 000477 1713279367.961134 mapread 0x011220 thru 0x019484 (0x8265 bytes) 000478 1713279367.961608 write 0x00490c thru 0x012983 (0xe078 bytes) 000481 1713279367.962685 trunc from 0x03a9e1 to 0x032a0a (0x7fd6 bytes) 000482 1713279367.966512 read 0x00a405 thru 0x00e44e (0x404a bytes) 000485 1713279367.967572 write 0x01c264 thru 0x021a38 (0x57d5 bytes) 000487 1713279367.968629 write 0x02d455 thru 0x031dd0 (0x497c bytes) 000488 1713279367.969404 write 0x0263cc thru 0x02de3d (0x7a72 bytes) 000489 1713279367.970275 trunc from 0x032a0b to 0x025c4f (0xcdbb bytes) 000490 1713279367.974086 write 0x03d449 thru 0x03ffff (0x2bb7 bytes) 000491 1713279367.974855 trunc from 0x040000 to 0x03ea85 (0x157a bytes) 000493 1713279367.978645 trunc from 0x03ea86 to 0x028d81 (0x15d04 bytes) 000494 1713279367.983075 MIRROR_EXTEND to 2 mirrors 000495 1713279368.090268 mapread 0x00ccb7 thru 0x00e9f5 (0x1d3f bytes) 000500 1713279368.105660 MIRROR_EXTEND to 3 mirrors 000501 1713279368.212975 MIRROR_RESYNC 3 mirrors 000502 1713279368.244563 mapwrite 0x03b8b1 thru 0x03ffff (0x474f bytes) 000503 1713279368.278133 mapread 0x039153 thru 0x03ffff (0x6ead bytes) 000506 1713279368.280170 MIRROR_EXTEND to 4 mirrors 000507 1713279368.386922 read 0x001287 thru 0x004249 (0x2fc3 bytes) 000508 1713279368.389309 trunc from 0x040000 to 0x019c51 (0x263ae bytes) 000510 1713279368.409229 mapread 0x00c0f8 thru 0x00c61d (0x526 bytes) 000511 1713279368.410900 write 0x03fe0c thru 0x03ffff (0x1f4 bytes) 000512 1713279368.411452 read 0x034f68 thru 0x036edb (0x1f74 bytes) 000513 1713279368.415571 MIRROR_RESYNC 4 mirrors 000515 1713279368.537287 read 0x02155c thru 0x02c866 (0xb30b bytes) 000516 1713279368.540418 read 0x009dc1 thru 0x016843 (0xca83 bytes) 000517 1713279368.543918 read 0x032d69 thru 0x03c0ec (0x9384 bytes) 000518 1713279368.546081 MIRROR_RESYNC 4 mirrors 000519 1713279368.594595 write 0x017671 thru 0x01f4ec (0x7e7c bytes) 000520 1713279368.610299 mapread 0x021963 thru 0x0316b4 (0xfd52 bytes) 000521 1713279368.616270 mapread 0x013ff1 thru 0x01eb4e (0xab5e bytes) 000522 1713279368.618568 trunc from 0x040000 to 0x01bc19 (0x243e6 bytes) 000523 1713279368.627178 trunc from 0x01bc1a to 0x03682b (0x1ac12 bytes) 000525 1713279368.631247 mapread 0x02c25d thru 0x03080e (0x45b2 bytes) 000526 1713279368.633881 write 0x019119 thru 0x0215cc (0x84b4 bytes) 000527 1713279368.635434 mapwrite 0x022e0f thru 0x030975 (0xdb67 bytes) 000528 1713279368.653158 read 0x01b834 thru 0x01e3fc (0x2bc9 bytes) 000529 1713279368.653552 mapwrite 0x0327dc thru 0x037966 (0x518b bytes) 000531 1713279368.672339 trunc from 0x037967 to 0x02747d (0x104e9 bytes) 000532 1713279368.675926 write 0x03599a thru 0x03ffff (0xa666 bytes) 000533 1713279368.676589 mapread 0x01a8e9 thru 0x026d30 (0xc448 bytes) 000534 1713279368.678143 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000537 1713279368.723537 mapwrite 0x004573 thru 0x012be0 (0xe66e bytes) 000538 1713279368.744054 read 0x01144c thru 0x013af2 (0x26a7 bytes) 000540 1713279368.744994 trunc from 0x040000 to 0x01a039 (0x25fc6 bytes) 000541 1713279368.749004 trunc from 0x01a03a to 0x02cc2e (0x12bf5 bytes) 000543 1713279368.752378 trunc from 0x02cc2f to 0x01431c (0x18912 bytes) 000544 1713279368.755548 trunc from 0x01431d to 0x00f2bf (0x505d bytes) 000548 1713279368.759437 write 0x014189 thru 0x023b98 (0xfa10 bytes) 000551 1713279368.760705 mapwrite 0x03bca5 thru 0x03ffff (0x435b bytes) 000553 1713279368.780843 trunc from 0x040000 to 0x021eaf (0x1e150 bytes) 000554 1713279368.784641 read 0x0114ca thru 0x0192d0 (0x7e07 bytes) 000555 1713279368.786950 trunc from 0x021eb0 to 0x036f94 (0x150e5 bytes) 000556 1713279368.790240 read 0x01427f thru 0x0181b3 (0x3f35 bytes) 000559 1713279368.792014 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000560 1713279368.833965 MIRROR_EXTEND to 3 mirrors 000562 1713279368.939136 write 0x01fe01 thru 0x02e4fd (0xe6fd bytes) 000564 1713279368.956619 trunc from 0x036f95 to 0x00c79f (0x2a7f5 bytes) 000565 1713279368.964794 mapread 0x00738d thru 0x007724 (0x398 bytes) 000566 1713279368.967450 trunc from 0x00c7a0 to 0x028ebf (0x1c720 bytes) 000567 1713279368.972385 MIRROR_RESYNC 3 mirrors 000568 1713279369.092217 trunc from 0x028ec0 to 0x028a0c (0x4b3 bytes) 000571 1713279369.108488 mapwrite 0x008a8d thru 0x014113 (0xb687 bytes) 000573 1713279369.129894 mapwrite 0x02b052 thru 0x02c62e (0x15dd bytes) 000574 1713279369.148993 trunc from 0x02c62f to 0x0089d5 (0x23c59 bytes) 000575 1713279369.152980 mapwrite 0x01699f thru 0x01b537 (0x4b99 bytes) 000576 1713279369.172250 trunc from 0x01b538 to 0x013837 (0x7d00 bytes) 000577 1713279369.175742 mapwrite 0x02b037 thru 0x036a5a (0xba24 bytes) 000579 1713279369.196977 read 0x029d83 thru 0x02c8a1 (0x2b1f bytes) 000581 1713279369.198489 read 0x0193c2 thru 0x01f720 (0x635f bytes) 000582 1713279369.201517 read 0x02376d thru 0x02e0f7 (0xa98b bytes) 000584 1713279369.202095 write 0x0089fc thru 0x00c43a (0x3a3f bytes) 000587 1713279369.203282 trunc from 0x036a5b to 0x035e19 (0xc41 bytes) 000588 1713279369.208515 MIRROR_RESYNC 3 mirrors 000589 1713279369.315710 write 0x025666 thru 0x02e42b (0x8dc6 bytes) 000590 1713279369.330063 mapread 0x00f208 thru 0x01debb (0xecb4 bytes) 000591 1713279369.334316 mapread 0x008ab4 thru 0x0112de (0x882b bytes) 000592 1713279369.337621 MIRROR_RESYNC 3 mirrors 000593 1713279369.445432 trunc from 0x035e1a to 0x0276da (0xe73f bytes) 000596 1713279369.463321 read 0x00ccfc thru 0x014609 (0x790e bytes) 000597 1713279369.465821 write 0x027fb3 thru 0x028a6d (0xabb bytes) 000598 1713279369.467387 trunc from 0x028a6e to 0x03cb0e (0x140a1 bytes) 000601 1713279369.472413 trunc from 0x03cb0f to 0x0275e9 (0x15525 bytes) 000602 1713279369.476457 MIRROR_RESYNC 3 mirrors 000603 1713279369.579884 mapwrite 0x000b43 thru 0x00a221 (0x96df bytes) 000604 1713279369.628910 write 0x02e015 thru 0x03bb11 (0xdafd bytes) 000605 1713279369.630222 mapwrite 0x02dc9f thru 0x037841 (0x9ba3 bytes) 000606 1713279369.648668 mapread 0x00b0e6 thru 0x00d8bb (0x27d6 bytes) 000607 1713279369.649192 mapread 0x0160a8 thru 0x01aa98 (0x49f1 bytes) 000610 1713279369.652576 write 0x003949 thru 0x00b970 (0x8028 bytes) 000611 1713279369.653364 mapwrite 0x02a3f8 thru 0x033119 (0x8d22 bytes) 000612 1713279369.669483 trunc from 0x03bb12 to 0x00e221 (0x2d8f0 bytes) 000613 1713279369.678273 trunc from 0x00e222 to 0x01bce0 (0xdabf bytes) 000615 1713279369.682017 mapwrite 0x018033 thru 0x024a25 (0xc9f3 bytes) 000616 1713279369.705261 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000617 1713279369.747004 mapwrite 0x0149c1 thru 0x015393 (0x9d3 bytes) 000620 1713279369.761783 mapread 0x015a14 thru 0x01612c (0x719 bytes) 000621 1713279369.765781 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000622 1713279369.793656 read 0x010c51 thru 0x013366 (0x2716 bytes) 000627 1713279369.797067 mapread 0x00230f thru 0x00560a (0x32fc bytes) 000628 1713279369.801736 write 0x03c809 thru 0x03ffff (0x37f7 bytes) 000629 1713279369.802591 write 0x0076f7 thru 0x01388b (0xc195 bytes) 000630 1713279369.803565 write 0x016b25 thru 0x01cb8b (0x6067 bytes) 000631 1713279369.806021 mapwrite 0x004bbc thru 0x004e0c (0x251 bytes) 000632 1713279369.820904 write 0x03ce3e thru 0x03ffff (0x31c2 bytes) 000633 1713279369.821472 mapread 0x026e81 thru 0x027a07 (0xb87 bytes) 000636 1713279369.824589 write 0x02f96e thru 0x038391 (0x8a24 bytes) 000642 1713279369.827595 write 0x032bc6 thru 0x038246 (0x5681 bytes) 000643 1713279369.828277 write 0x004747 thru 0x00531d (0xbd7 bytes) 000644 1713279369.828952 mapread 0x0364df thru 0x03b659 (0x517b bytes) 000647 1713279369.831409 read 0x021227 thru 0x0266a7 (0x5481 bytes) 000648 1713279369.831678 trunc from 0x040000 to 0x038842 (0x77bd bytes) 000649 1713279369.835467 mapread 0x00fad0 thru 0x01e6d2 (0xec03 bytes) 000650 1713279369.836065 mapwrite 0x001fb1 thru 0x006590 (0x45e0 bytes) 000651 1713279369.850123 read 0x010187 thru 0x01b662 (0xb4dc bytes) 000653 1713279369.850816 read 0x0098ae thru 0x00cf1d (0x3670 bytes) 000656 1713279369.851993 mapread 0x013842 thru 0x01a5c0 (0x6d7f bytes) 000657 1713279369.852404 write 0x03359b thru 0x03ffff (0xca65 bytes) 000660 1713279369.853459 mapread 0x020ba1 thru 0x02de42 (0xd2a2 bytes) 000663 1713279369.854785 write 0x00cdfd thru 0x015edc (0x90e0 bytes) 000665 1713279369.855533 trunc from 0x040000 to 0x0161fc (0x29e03 bytes) 000668 1713279369.860048 read 0x01371d thru 0x0161fc (0x2ae0 bytes) 000669 1713279369.860387 read 0x006c74 thru 0x00dbbd (0x6f4a bytes) 000670 1713279369.860646 trunc from 0x0161fd to 0x0107fd (0x59ff bytes) 000671 1713279369.864309 write 0x03bd0a thru 0x03ffff (0x42f6 bytes) 000672 1713279369.865201 read 0x005056 thru 0x00a447 (0x53f2 bytes) 000673 1713279369.865628 read 0x03a1b6 thru 0x03a3db (0x226 bytes) 000675 1713279369.869014 mapwrite 0x01bc23 thru 0x01d3fd (0x17db bytes) 000676 1713279369.883553 mapread 0x025901 thru 0x031c6c (0xc36c bytes) 000677 1713279369.884396 mapread 0x015e4e thru 0x01ae89 (0x503c bytes) 000678 1713279369.885091 write 0x001928 thru 0x00adf7 (0x94d0 bytes) 000680 1713279369.886045 write 0x03f3cd thru 0x03ffff (0xc33 bytes) 000686 1713279369.888795 write 0x02e651 thru 0x037d78 (0x9728 bytes) 000688 1713279369.890072 mapread 0x03e826 thru 0x03ffff (0x17da bytes) 000690 1713279369.891107 mapwrite 0x00adab thru 0x0171c3 (0xc419 bytes) 000694 1713279369.907417 write 0x00e67c thru 0x0194b9 (0xae3e bytes) 000698 1713279369.909279 write 0x004b1d thru 0x006282 (0x1766 bytes) 000699 1713279369.909921 read 0x016f73 thru 0x017d21 (0xdaf bytes) 000700 1713279369.910292 mapread 0x022bdc thru 0x02c88a (0x9caf bytes) 000704 1713279369.911661 mapread 0x02b7ff thru 0x02f06a (0x386c bytes) 000705 1713279369.913544 MIRROR_EXTEND to 2 mirrors 000710 1713279370.023972 read 0x029120 thru 0x036c6c (0xdb4d bytes) 000711 1713279370.027297 read 0x02c268 thru 0x039098 (0xce31 bytes) 000714 1713279370.034155 MIRROR_EXTEND to 3 mirrors 000717 1713279370.149203 mapwrite 0x004a83 thru 0x00d9a6 (0x8f24 bytes) 000718 1713279370.192841 mapread 0x02b809 thru 0x02ceb9 (0x16b1 bytes) 000719 1713279370.195482 read 0x016681 thru 0x018713 (0x2093 bytes) 000720 1713279370.197373 write 0x002e4d thru 0x01125d (0xe411 bytes) 000721 1713279370.199094 mapread 0x002e05 thru 0x00cc16 (0x9e12 bytes) 000725 1713279370.200230 write 0x00e8de thru 0x01c20d (0xd930 bytes) 000726 1713279370.202009 mapread 0x00c11f thru 0x013c78 (0x7b5a bytes) 000728 1713279370.202650 mapwrite 0x011a73 thru 0x017fd4 (0x6562 bytes) 000729 1713279370.215230 write 0x037bde thru 0x039447 (0x186a bytes) 000732 1713279370.217900 mapread 0x00b8ee thru 0x00c848 (0xf5b bytes) 000734 1713279370.218584 write 0x00197b thru 0x005855 (0x3edb bytes) 000738 1713279370.219704 mapwrite 0x0088a1 thru 0x00b4d0 (0x2c30 bytes) 000739 1713279370.233774 MIRROR_RESYNC 3 mirrors 000741 1713279370.333576 MIRROR_RESYNC 3 mirrors 000744 1713279370.371278 mapwrite 0x022224 thru 0x023087 (0xe64 bytes) 000745 1713279370.416753 trunc from 0x040000 to 0x0122d7 (0x2dd28 bytes) 000746 1713279370.424912 write 0x00e6a0 thru 0x010580 (0x1ee1 bytes) 000748 1713279370.427461 mapwrite 0x032684 thru 0x03ecda (0xc657 bytes) 000749 1713279370.450645 mapread 0x020d05 thru 0x02455a (0x3856 bytes) 000750 1713279370.454694 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000752 1713279370.497437 write 0x0350fd thru 0x03ffff (0xaf03 bytes) 000753 1713279370.499308 mapwrite 0x01d66e thru 0x029566 (0xbef9 bytes) 000755 1713279370.522572 MIRROR_EXTEND to 3 mirrors 000757 1713279370.628547 mapread 0x03122d thru 0x03e167 (0xcf3b bytes) 000759 1713279370.637319 write 0x00879e thru 0x01574f (0xcfb2 bytes) 000760 1713279370.654255 write 0x03a08e thru 0x03d81c (0x378f bytes) 000762 1713279370.656546 write 0x028a45 thru 0x0319bb (0x8f77 bytes) 000763 1713279370.659787 read 0x03ea25 thru 0x03ffff (0x15db bytes) 000764 1713279370.660088 trunc from 0x040000 to 0x03f27c (0xd83 bytes) 000765 1713279370.668113 write 0x014c89 thru 0x01863d (0x39b5 bytes) 000769 1713279370.671267 mapwrite 0x004cd5 thru 0x00580f (0xb3b bytes) 000771 1713279370.687414 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000772 1713279370.730631 read 0x02f6ff thru 0x03ae65 (0xb767 bytes) 000775 1713279370.733590 write 0x036629 thru 0x036aae (0x486 bytes) 000780 1713279370.735525 read 0x00d3e3 thru 0x01938e (0xbfac bytes) 000781 1713279370.740287 MIRROR_RESYNC 2 mirrors 000782 1713279370.831420 trunc from 0x03f27d to 0x0255ec (0x19c90 bytes) 000784 1713279370.849140 trunc from 0x0255ed to 0x00fa87 (0x15b65 bytes) 000788 1713279370.854528 mapread 0x001219 thru 0x00ca0e (0xb7f6 bytes) 000789 1713279370.860600 trunc from 0x00fa88 to 0x00f211 (0x876 bytes) 000790 1713279370.864236 mapwrite 0x02dd76 thru 0x0313b4 (0x363f bytes) 000791 1713279370.884643 mapwrite 0x032cae thru 0x03fd98 (0xd0eb bytes) 000793 1713279370.906377 mapread 0x025e72 thru 0x03015a (0xa2e9 bytes) 000795 1713279370.909409 trunc from 0x03fd99 to 0x017132 (0x28c66 bytes) 000797 1713279370.913308 trunc from 0x017133 to 0x039e95 (0x22d63 bytes) 000798 1713279370.916641 trunc from 0x039e96 to 0x03c478 (0x25e3 bytes) 000800 1713279370.919899 trunc from 0x03c479 to 0x035ee3 (0x6595 bytes) 000804 1713279370.925916 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000807 1713279370.955482 write 0x012868 thru 0x014522 (0x1cbb bytes) 000808 1713279370.958226 trunc from 0x035ee4 to 0x014ad5 (0x2140e bytes) 000812 1713279370.962258 mapread 0x010358 thru 0x014ad5 (0x477e bytes) 000813 1713279370.964571 mapread 0x00a7f9 thru 0x014ad5 (0xa2dd bytes) 000814 1713279370.965185 read 0x00e4e4 thru 0x014ad5 (0x65f2 bytes) 000818 1713279370.966559 read 0x0129e2 thru 0x014ad5 (0x20f4 bytes) 000824 1713279370.968255 write 0x0233c8 thru 0x033100 (0xfd39 bytes) 000825 1713279370.969273 mapread 0x02c467 thru 0x030d03 (0x489d bytes) 000826 1713279370.969816 trunc from 0x033101 to 0x039720 (0x6620 bytes) 000827 1713279370.973611 read 0x00f4e7 thru 0x0146ab (0x51c5 bytes) 000828 1713279370.973985 read 0x017333 thru 0x024239 (0xcf07 bytes) 000831 1713279370.977889 mapwrite 0x0032ee thru 0x00cc09 (0x991c bytes) 000832 1713279370.994291 trunc from 0x039721 to 0x0365c7 (0x3159 bytes) 000840 1713279370.999901 mapread 0x009baa thru 0x00dacd (0x3f24 bytes) 000841 1713279371.000343 read 0x025417 thru 0x033e1e (0xea08 bytes) 000842 1713279371.000640 mapwrite 0x019342 thru 0x01b86f (0x252e bytes) 000845 1713279371.013355 mapwrite 0x0126e2 thru 0x01a5de (0x7efd bytes) 000846 1713279371.026062 mapread 0x013f38 thru 0x01c7d4 (0x889d bytes) 000847 1713279371.026478 trunc from 0x0365c8 to 0x002666 (0x33f61 bytes) 000849 1713279371.031372 write 0x027a33 thru 0x02d10f (0x56dd bytes) 000850 1713279371.032195 read 0x00c93c thru 0x013dc5 (0x748a bytes) 000851 1713279371.034699 read 0x0136c8 thru 0x015bd5 (0x250e bytes) 000854 1713279371.035397 mapread 0x029a6b thru 0x02d10f (0x36a5 bytes) 000859 1713279371.036822 mapread 0x00ed53 thru 0x01c251 (0xd4ff bytes) 000860 1713279371.037258 mapread 0x009eec thru 0x011794 (0x78a9 bytes) 000862 1713279371.037913 read 0x005b62 thru 0x007fb4 (0x2453 bytes) 000863 1713279371.038166 read 0x020ffc thru 0x02405d (0x3062 bytes) 000864 1713279371.038418 write 0x006f86 thru 0x00b251 (0x42cc bytes) 000867 1713279371.039283 write 0x0348c8 thru 0x036831 (0x1f6a bytes) 000869 1713279371.039930 trunc from 0x036832 to 0x032f5b (0x38d6 bytes) 000870 1713279371.043582 read 0x02c2a2 thru 0x032f5b (0x6cba bytes) 000871 1713279371.045705 read 0x002638 thru 0x010328 (0xdcf1 bytes) 000873 1713279371.046276 trunc from 0x032f5c to 0x0037ca (0x2f791 bytes) 000875 1713279371.050426 mapread 0x00065a thru 0x0037ca (0x3171 bytes) 000877 1713279371.051326 mapwrite 0x029a18 thru 0x035e31 (0xc41a bytes) 000878 1713279371.073556 mapread 0x03127b thru 0x035e31 (0x4bb7 bytes) 000879 1713279371.074163 mapwrite 0x01c2be thru 0x0269a0 (0xa6e3 bytes) 000880 1713279371.088695 mapwrite 0x036795 thru 0x03b9b2 (0x521e bytes) 000881 1713279371.108402 trunc from 0x03b9b3 to 0x02cbca (0xede8 bytes) 000882 1713279371.112179 mapread 0x00a14b thru 0x015f23 (0xbdd9 bytes) 000883 1713279371.112709 trunc from 0x02cbcb to 0x00b007 (0x21bc3 bytes) 000888 1713279371.117547 read 0x007cb3 thru 0x00b007 (0x3355 bytes) 000889 1713279371.117884 mapwrite 0x023e57 thru 0x02e831 (0xa9db bytes) 000894 1713279371.141660 read 0x0075e2 thru 0x015bda (0xe5f9 bytes) 000895 1713279371.142046 read 0x027b02 thru 0x02e831 (0x6d30 bytes) 000896 1713279371.142299 mapread 0x00f364 thru 0x01638e (0x702b bytes) 000897 1713279371.142941 mapwrite 0x0124f5 thru 0x020f6b (0xea77 bytes) 000898 1713279371.157461 mapread 0x0030f1 thru 0x00e3df (0xb2ef bytes) 000899 1713279371.157970 read 0x0272d8 thru 0x029adc (0x2805 bytes) 000900 1713279371.158230 trunc from 0x02e832 to 0x034f96 (0x6765 bytes) 000901 1713279371.162070 trunc from 0x034f97 to 0x0012ee (0x33ca8 bytes) 000902 1713279371.165526 mapwrite 0x00e78a thru 0x01b380 (0xcbf7 bytes) 000904 1713279371.185121 trunc from 0x01b381 to 0x029a9f (0xe71f bytes) 000905 1713279371.188548 mapwrite 0x00426e thru 0x00ace7 (0x6a7a bytes) 000909 1713279371.204382 mapread 0x01060f thru 0x01ec69 (0xe65b bytes) 000912 1713279371.207949 mapread 0x010cb3 thru 0x018713 (0x7a61 bytes) 000914 1713279371.209802 MIRROR_EXTEND to 2 mirrors 000915 1713279371.321789 mapwrite 0x00f5d1 thru 0x019823 (0xa253 bytes) 000916 1713279371.374002 MIRROR_RESYNC 2 mirrors 000919 1713279371.461330 mapwrite 0x00ba1e thru 0x016352 (0xa935 bytes) 000921 1713279371.505713 mapwrite 0x02d146 thru 0x03402e (0x6ee9 bytes) 000923 1713279371.531853 trunc from 0x03402f to 0x027e37 (0xc1f7 bytes) 000927 1713279371.536290 mapread 0x0151c1 thru 0x01bcb8 (0x6af8 bytes) 000930 1713279371.537322 mapread 0x00fe35 thru 0x01f99d (0xfb69 bytes) 000932 1713279371.540153 read 0x022569 thru 0x027e37 (0x58cf bytes) 000933 1713279371.540566 write 0x03fa9c thru 0x03ffff (0x564 bytes) 000934 1713279371.542419 MIRROR_EXTEND to 3 mirrors 000937 1713279371.656063 trunc from 0x040000 to 0x031a15 (0xe5ea bytes) 000938 1713279371.674170 mapread 0x0074be thru 0x00cfea (0x5b2d bytes) 000940 1713279371.679251 mapread 0x01ac0d thru 0x0241f6 (0x95ea bytes) 000941 1713279371.684348 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000942 1713279371.727147 read 0x023da4 thru 0x02f863 (0xbac0 bytes) 000943 1713279371.729753 trunc from 0x031a16 to 0x01e00a (0x13a0b bytes) 000945 1713279371.734778 MIRROR_RESYNC 2 mirrors 000946 1713279371.820964 trunc from 0x01e00b to 0x02914e (0xb144 bytes) 000947 1713279371.839546 MIRROR_EXTEND to 3 mirrors 000949 1713279371.941218 write 0x03ff59 thru 0x03ffff (0xa7 bytes) 000950 1713279371.955157 mapread 0x0023e7 thru 0x00b59b (0x91b5 bytes) 000953 1713279371.960177 write 0x01e050 thru 0x02665a (0x860b bytes) 000955 1713279371.964155 MIRROR_EXTEND to 4 mirrors 000956 1713279372.083560 trunc from 0x040000 to 0x01e148 (0x21eb7 bytes) 000957 1713279372.098460 trunc from 0x01e149 to 0x002317 (0x1be31 bytes) 000960 1713279372.102801 mapread 0x000ee2 thru 0x002317 (0x1436 bytes) 000962 1713279372.106691 trunc from 0x002318 to 0x006f5d (0x4c46 bytes) 000964 1713279372.110771 read 0x0046a2 thru 0x006f5d (0x28bc bytes) 000965 1713279372.112445 write 0x0135ef thru 0x014769 (0x117b bytes) 000966 1713279372.112985 mapread 0x00c807 thru 0x014492 (0x7c8c bytes) 000968 1713279372.116731 MIRROR_RESYNC 4 mirrors 000969 1713279372.243968 read 0x00fec2 thru 0x014769 (0x48a8 bytes) 000970 1713279372.246041 mapread 0x00271b thru 0x007f2f (0x5815 bytes) 000971 1713279372.250218 write 0x01f3b8 thru 0x01f849 (0x492 bytes) 000973 1713279372.263376 mapread 0x014e52 thru 0x01f849 (0xa9f8 bytes) 000974 1713279372.268859 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000976 1713279372.315680 write 0x00bde1 thru 0x013106 (0x7326 bytes) 000977 1713279372.318544 write 0x01d78c thru 0x02c0ff (0xe974 bytes) 000978 1713279372.322751 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000980 1713279372.366579 trunc from 0x02c100 to 0x0228a3 (0x985c bytes) 000984 1713279372.375241 MIRROR_RESYNC 2 mirrors 000988 1713279372.458293 mapread 0x01b6b7 thru 0x022606 (0x6f50 bytes) 000989 1713279372.462591 mapread 0x00ce09 thru 0x0149a7 (0x7b9f bytes) 000990 1713279372.466109 read 0x007468 thru 0x009671 (0x220a bytes) 000991 1713279372.467207 MIRROR_EXTEND to 3 mirrors 000992 1713279372.581966 read 0x016960 thru 0x01915a (0x27fb bytes) 000993 1713279372.583507 read 0x01d10d thru 0x01e34a (0x123e bytes) 000994 1713279372.585865 trunc from 0x0228a4 to 0x00e8ec (0x13fb7 bytes) 000996 1713279372.604994 mapread 0x000e4f thru 0x002c4c (0x1dfe bytes) 000997 1713279372.608849 trunc from 0x00e8ed to 0x010491 (0x1ba5 bytes) 000999 1713279372.613489 read 0x00b3e8 thru 0x00d3ed (0x2006 bytes) 001000 1713279372.615861 write 0x022aa9 thru 0x02b492 (0x89ea bytes) All operations completed A-OK! PASS 70a (10s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 10:56:15 (1713279375) Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3055552 .. Extending file size to 8024576 .. done Extending file size to 3559840 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7178592 .. Extending file size to 8887232 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8542208 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9031936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8961120 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9053408 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9155552 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 6754 6755 6757 6758 6761 failed 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:57:20 (1713279440) Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6651456 .. Extending file size to 6879776 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7740256 .. Extending file size to 7822400 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7726848 .. Extending file size to 7845056 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8886944 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9267840 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8252512 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9113792 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9310720 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 10136 10137 10138 10140 10142 done 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:58:26 (1713279506) Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0324817 s, 64.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0769155 s, 27.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:58:32 (1713279512) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0253543 s, 82.7 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 10:58:35 (1713279515) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xccc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xccc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xccd:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcc7:0x0] } PASS 203 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:58:38 (1713279518) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0293202 s, 71.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:58:41 (1713279521) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0272921 s, 76.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xccf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcca:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xccf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcca:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xccb:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:58:44 (1713279524) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0245294 s, 85.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xccc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xccc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xccd:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:59:09 (1713279549) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163805 s, 64.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xccf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xccf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd3:0x0] } PASS 204d (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:59:52 (1713279592) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcd0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcd1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0243821 s, 86.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcd0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcd1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcd0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcd1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (42s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:00:36 (1713279636) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. Creating new pool oleg203-server: Pool lustre.test_204f created Adding targets to pool oleg203-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg203-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0264577 s, 79.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcd2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcd2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg203-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg203-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg203-server: Pool lustre.test_204f destroyed PASS 204f (52s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 11:01:30 (1713279690) PASS 205 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:01:33 (1713279693) oleg203-server: Pool lustre.test_206 created Adding targets to pool oleg203-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg203-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg203-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg203-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg203-server: Pool lustre.test_206 destroyed PASS 206 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:01:47 (1713279707) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181725 s, 57.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00933013 s, 112 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xb163b263 Mirror 3: 0xb163b263 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:01:51 (1713279711) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:01:54 (1713279714) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:01:56 (1713279716) Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0164363 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1330000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1bf0000 Read this (4 bytes): BBBB 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:02:11 (1713279731) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00272142 s, 3.0 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 11:02:14 (1713279734) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00806945 s, 130 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6bf lmm_fid: [0x200000404:0x6bf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6bf lmm_fid: [0x200000404:0x6bf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcdc:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835647 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-16 11:02:15.000000000 -0400 Modify: 2024-04-16 11:02:15.000000000 -0400 Change: 2024-04-16 11:02:15.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:02:18 (1713279738) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 765 sec ============ 11:02:19 (1713279739) === sanity-flr: start cleanup 11:02:20 (1713279740) === === sanity-flr: finish cleanup 11:02:20 (1713279740) ===