-----============= acceptance-small: sanity-flr ============----- Fri Apr 19 08:49:36 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:49:38 (1713530978) === oleg238-client.virtnet: executing check_config_client /mnt/lustre oleg238-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg238-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a6ea000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a6ea000.idle_timeout=debug disable quota as required oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:49:44 (1713530984) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6726] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 08:49:45 (1713530985) PASS 0a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:49:50 (1713530990) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:49:52 (1713530992) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 08:49:55 (1713530995) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:50:00 (1713531000) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:50:02 (1713531002) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 08:50:04 (1713531004) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a6ea000.stats=clear osc.lustre-OST0001-osc-ffff88012a6ea000.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:50:08 (1713531008) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:50:12 (1713531012) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:50:15 (1713531015) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 08:50:18 (1713531018) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 08:50:21 (1713531021) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:50:25 (1713531025) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:50:27 (1713531027) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:50:30 (1713531030) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 08:50:34 (1713531034) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:50:37 (1713531037) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 0.641566 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0476123 s, 22.0 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:50:42 (1713531042) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 0.0932532 s, 191 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:50:45 (1713531045) Stopping /mnt/lustre-ost1 (opts:) on oleg238-server oleg238-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a6ea000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a6ea000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg238-server: oleg238-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg238-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg238-server oleg238-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a6ea000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a6ea000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg238-server: oleg238-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg238-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:51:08 (1713531068) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0709365 s, 73.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg238-server oleg238-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a6ea000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a6ea000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg238-server: oleg238-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg238-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:51:19 (1713531079) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg238-server oleg238-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a6ea000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a6ea000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg238-server: oleg238-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg238-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg238-server oleg238-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a6ea000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a6ea000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg238-server: oleg238-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg238-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:51:34 (1713531094) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.685064 s, 76.5 MB/s -rw-r--r-- 1 root root 52428800 Apr 19 08:51 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.167s user 0m0.001s sys 0m0.098s Stopping /mnt/lustre-ost1 (opts:) on oleg238-server oleg238-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a6ea000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a6ea000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.182s user 0m0.000s sys 0m0.099s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg238-server: oleg238-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg238-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg238-server oleg238-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a6ea000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a6ea000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.193s user 0m0.000s sys 0m0.107s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg238-server: oleg238-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg238-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a6ea000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a6ea000.ost_server_uuid in FULL state after 0 sec PASS 33b (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:51:50 (1713531110) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:51:52 (1713531112) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:51:54 (1713531114) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 08:51:56 (1713531116) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 08:51:59 (1713531119) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.117403 s, 107 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:52:03 (1713531123) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00846828 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00735077 s, 143 MB/s PASS 36b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:52:06 (1713531126) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:52:10 (1713531130) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000513105 s, 2.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000435218 s, 2.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0005941 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000356235 s, 2.9 MB/s flv=10 olv=10 PASS 36d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 08:52:13 (1713531133) create test files with size 15732210 .. 945b9dbcadf31c78ed8d8992b5eecfc6 - 411c312dab64b06046c5a62f9e311b7d - 107d5c28566209d56579b51c01e3b05d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg238-client.virtnet /mnt/lustre (opts:) Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 08:52:19 (1713531139) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:52:23 (1713531143) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:52:25 (1713531145) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0235902 s, 44.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x49:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00602361 s, 174 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 08:52:28 (1713531148) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0225599 s, 93.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.076194 s, 55.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.032254 s, 65.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 08:52:32 (1713531152) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 08:52:34 (1713531154) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:52:36 (1713531156) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0063119 s, 166 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 08:52:40 (1713531160) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:52:42 (1713531162) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:52:45 (1713531165) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0598479 s, 175 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg238-client.virtnet /mnt/lustre (opts:) Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:52:49 (1713531169) llite.lustre-ffff8800b63f0000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0541897 s, 194 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg238-client.virtnet /mnt/lustre (opts:) Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a744e800.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:52:53 (1713531173) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 08:53:01 (1713531181) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 08:53:07 (1713531187) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 08:53:09 (1713531189) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0541892 s, 58.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0428944 s, 73.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: a2c4db9f8eb3db27cf34d8fe5365b6bf - mirror 1 checksum: 0d9cc2d800648f1d619e875eff18198e - mirror 2 checksum: a2c4db9f8eb3db27cf34d8fe5365b6bf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 19 08:53:11 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 19 08:53:11 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a2c4db9f8eb3db27cf34d8fe5365b6bf - mirror 1 checksum: 0d9cc2d800648f1d619e875eff18198e - mirror 2 checksum: 0d9cc2d800648f1d619e875eff18198e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 19 08:53:11 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 19 08:53:11 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 48 (3s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:53:13 (1713531193) Failing mds1 on oleg238-server Stopping /mnt/lustre-mds1 (opts:) on oleg238-server 08:53:15 (1713531195) shut down Failover mds1 to oleg238-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 08:53:27 (1713531207) targets are mounted 08:53:27 (1713531207) facet_failover done oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100315 s, 105 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 50A (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:53:34 (1713531214) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 08:53:36 (1713531216) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:53:38 (1713531218) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00225501 s, 7.3 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 08:53:41 (1713531221) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:53:44 (1713531224) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:53:46 (1713531226) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:53:51 (1713531231) create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] PASS 61a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:54:10 (1713531250) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:54:28 (1713531268) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:54:46 (1713531286) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 08:54:49 (1713531289) mirror create pid 26207 mirror split pid 26208 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4661: 26207 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4661: 26208 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 08:55:52 (1713531352) Seed set to 1353 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x35c77 000001 1713531353.348776 trunc from 00000000 to 0x035c76 (0x35c77 bytes) 000002 1713531353.352442 write 0x00d7ee thru 0x01ae49 (0xd65c bytes) 000003 1713531353.355019 write 0x004659 thru 0x00cf14 (0x88bc bytes) 000007 1713531353.359895 write 0x01b3c7 thru 0x025459 (0xa093 bytes) 000008 1713531353.362781 mapread 0x02e613 thru 0x0319e7 (0x33d5 bytes) 000011 1713531353.365279 trunc from 0x035c77 to 0x00e535 (0x27741 bytes) 000012 1713531353.368748 trunc from 0x00e536 to 0x0111cf (0x2c9a bytes) 000016 1713531353.372802 mapread 0x01082f thru 0x0111cf (0x9a1 bytes) 000017 1713531353.374711 mapread 0x00d299 thru 0x0111cf (0x3f37 bytes) 000018 1713531353.375307 mapwrite 0x029baf thru 0x02f3b5 (0x5807 bytes) 000019 1713531353.411533 mapwrite 0x03319d thru 0x036b97 (0x39fb bytes) 000020 1713531353.431710 write 0x031b36 thru 0x03579b (0x3c66 bytes) 000021 1713531353.432291 read 0x018222 thru 0x02559d (0xd37c bytes) 000022 1713531353.434905 trunc from 0x036b98 to 0x017f37 (0x1ec60 bytes) 000023 1713531353.438554 mapwrite 0x039bf9 thru 0x03ffff (0x6407 bytes) 000025 1713531353.460309 mapwrite 0x036edd thru 0x03ffff (0x9123 bytes) 000026 1713531353.475307 trunc from 0x040000 to 0x0335e0 (0xca1f bytes) 000027 1713531353.479023 write 0x027524 thru 0x031228 (0x9d05 bytes) 000028 1713531353.482710 read 0x02f9d5 thru 0x0335e0 (0x3c0c bytes) 000029 1713531353.483139 read 0x0238e2 thru 0x031c3d (0xe35c bytes) 000032 1713531353.484226 mapread 0x02fc59 thru 0x0335e0 (0x3988 bytes) 000034 1713531353.485150 trunc from 0x0335e1 to 0x0327d0 (0xe10 bytes) 000036 1713531353.488905 write 0x007c39 thru 0x00f075 (0x743d bytes) 000037 1713531353.489436 mapread 0x031952 thru 0x0327d0 (0xe7f bytes) 000038 1713531353.489993 mapread 0x022b87 thru 0x02e3fb (0xb875 bytes) 000042 1713531353.491080 trunc from 0x0327d1 to 0x019f6a (0x18866 bytes) 000043 1713531353.494772 mapread 0x0005b8 thru 0x00b8e8 (0xb331 bytes) 000044 1713531353.498066 MIRROR_EXTEND to 2 mirrors 000045 1713531353.585084 mapread 0x007101 thru 0x0077ad (0x6ad bytes) 000048 1713531353.586998 mapread 0x0025dc thru 0x00d7e6 (0xb20b bytes) 000050 1713531353.591974 MIRROR_RESYNC 2 mirrors 000052 1713531353.612836 mapread 0x010ec0 thru 0x013470 (0x25b1 bytes) truncating to largest ever: 0x3b1ea 000053 1713531353.613468 trunc from 0x019f6b to 0x03b1e9 (0x2127f bytes) 000054 1713531353.633228 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000061 1713531353.667967 write 0x0189e9 thru 0x027d2a (0xf342 bytes) 000062 1713531353.671634 read 0x0050d0 thru 0x00c803 (0x7734 bytes) 000063 1713531353.674286 trunc from 0x03b1ea to 0x02b5c1 (0xfc28 bytes) 000065 1713531353.679732 MIRROR_EXTEND to 2 mirrors 000066 1713531353.787790 write 0x034e78 thru 0x03ffff (0xb188 bytes) 000067 1713531353.802349 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000070 1713531353.833419 mapwrite 0x03d0ce thru 0x03f9e5 (0x2918 bytes) 000075 1713531353.851046 mapwrite 0x001afa thru 0x006fce (0x54d5 bytes) 000076 1713531353.865614 mapwrite 0x01f246 thru 0x02eec9 (0xfc84 bytes) 000077 1713531353.882616 read 0x003631 thru 0x011bb4 (0xe584 bytes) 000082 1713531353.885199 read 0x03440e thru 0x03ffff (0xbbf2 bytes) 000083 1713531353.887748 read 0x023c26 thru 0x02ab0c (0x6ee7 bytes) 000084 1713531353.888213 trunc from 0x040000 to 0x012994 (0x2d66b bytes) 000085 1713531353.894954 write 0x015247 thru 0x020473 (0xb22d bytes) 000087 1713531353.895808 write 0x028e25 thru 0x028e4c (0x28 bytes) 000088 1713531353.896157 write 0x006632 thru 0x00d062 (0x6a31 bytes) 000090 1713531353.896895 trunc from 0x028e4d to 0x031cc2 (0x8e76 bytes) 000092 1713531353.900514 write 0x018e35 thru 0x0252b6 (0xc482 bytes) 000093 1713531353.903541 trunc from 0x031cc3 to 0x039033 (0x7371 bytes) 000094 1713531353.907194 write 0x01b23f thru 0x01c311 (0x10d3 bytes) 000096 1713531353.908055 read 0x01a2f6 thru 0x025729 (0xb434 bytes) 000097 1713531353.908461 read 0x027cc5 thru 0x0296db (0x1a17 bytes) 000100 1713531353.911542 write 0x02c1f1 thru 0x03b2e1 (0xf0f1 bytes) 000102 1713531353.912354 write 0x02dd18 thru 0x03ab60 (0xce49 bytes) 000103 1713531353.912973 write 0x02a87c thru 0x02d2ef (0x2a74 bytes) 000105 1713531353.913604 mapread 0x03af75 thru 0x03b2e1 (0x36d bytes) 000107 1713531353.914182 mapwrite 0x02b976 thru 0x03834d (0xc9d8 bytes) 000108 1713531353.931330 write 0x029dea thru 0x035343 (0xb55a bytes) 000113 1713531353.933503 write 0x01f191 thru 0x02bf0c (0xcd7c bytes) truncating to largest ever: 0x3b9f3 000114 1713531353.934426 trunc from 0x03b2e2 to 0x03b9f2 (0x711 bytes) 000116 1713531353.939174 mapread 0x0245b2 thru 0x027e43 (0x3892 bytes) 000118 1713531353.940183 mapread 0x008ddc thru 0x015e70 (0xd095 bytes) 000120 1713531353.941323 mapread 0x00b247 thru 0x015e23 (0xabdd bytes) 000121 1713531353.941795 mapwrite 0x01c719 thru 0x01e8f0 (0x21d8 bytes) 000122 1713531353.957146 write 0x02530e thru 0x0347b7 (0xf4aa bytes) 000123 1713531353.957993 write 0x02826d thru 0x037561 (0xf2f5 bytes) 000125 1713531353.960380 MIRROR_EXTEND to 2 mirrors 000126 1713531354.070137 write 0x01f3fa thru 0x02b56b (0xc172 bytes) 000127 1713531354.087429 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000129 1713531354.115312 mapread 0x02aa7f thru 0x0364c2 (0xba44 bytes) 000131 1713531354.122060 write 0x01455e thru 0x0156dc (0x117f bytes) 000132 1713531354.124227 write 0x00756d thru 0x00d2a0 (0x5d34 bytes) 000134 1713531354.127678 MIRROR_EXTEND to 2 mirrors 000136 1713531354.241246 trunc from 0x03b9f3 to 0x0264aa (0x15548 bytes) 000139 1713531354.260049 mapread 0x01500f thru 0x01cfab (0x7f9d bytes) 000140 1713531354.264729 mapread 0x010f4b thru 0x019297 (0x834d bytes) 000142 1713531354.267323 trunc from 0x0264ab to 0x01e175 (0x8335 bytes) 000143 1713531354.272121 MIRROR_RESYNC 2 mirrors 000144 1713531354.352878 write 0x0381b2 thru 0x03ffff (0x7e4e bytes) 000146 1713531354.366880 write 0x024684 thru 0x028a0a (0x4387 bytes) 000147 1713531354.369607 mapread 0x02cd1b thru 0x0371c6 (0xa4ac bytes) 000148 1713531354.373639 trunc from 0x040000 to 0x008638 (0x379c7 bytes) 000149 1713531354.382060 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000150 1713531354.410730 trunc from 0x008639 to 0x037bec (0x2f5b4 bytes) 000151 1713531354.414568 mapread 0x01ce86 thru 0x021dac (0x4f27 bytes) 000155 1713531354.420175 read 0x025886 thru 0x02a6bf (0x4e3a bytes) 000158 1713531354.422424 write 0x0026ab thru 0x00bdba (0x9710 bytes) 000159 1713531354.425039 write 0x016836 thru 0x016ffb (0x7c6 bytes) 000160 1713531354.427389 mapread 0x01ce0e thru 0x0292a2 (0xc495 bytes) 000161 1713531354.428122 write 0x01da70 thru 0x027c32 (0xa1c3 bytes) 000163 1713531354.429328 mapread 0x010dc2 thru 0x0182f6 (0x7535 bytes) 000164 1713531354.430004 read 0x03634a thru 0x0363cc (0x83 bytes) 000165 1713531354.432954 trunc from 0x037bed to 0x02f879 (0x8373 bytes) 000167 1713531354.437017 mapwrite 0x028441 thru 0x02df53 (0x5b13 bytes) truncating to largest ever: 0x3c7d7 000168 1713531354.452672 trunc from 0x02f87a to 0x03c7d6 (0xcf5d bytes) 000170 1713531354.456452 mapread 0x0156e4 thru 0x020ace (0xb3eb bytes) 000171 1713531354.456929 read 0x004626 thru 0x007ceb (0x36c6 bytes) 000172 1713531354.457222 write 0x01b4fb thru 0x01d37e (0x1e84 bytes) 000174 1713531354.457826 read 0x019866 thru 0x0205c7 (0x6d62 bytes) 000175 1713531354.458067 read 0x0270ad thru 0x0348d7 (0xd82b bytes) 000176 1713531354.460309 mapread 0x03b077 thru 0x03c7d6 (0x1760 bytes) 000177 1713531354.460759 trunc from 0x03c7d7 to 0x02d626 (0xf1b0 bytes) 000183 1713531354.465503 write 0x0082f1 thru 0x00d0ba (0x4dca bytes) 000187 1713531354.466643 mapwrite 0x011910 thru 0x01cc8d (0xb37e bytes) 000190 1713531354.483315 MIRROR_EXTEND to 2 mirrors 000191 1713531354.576409 read 0x007d76 thru 0x00842f (0x6ba bytes) 000195 1713531354.578241 trunc from 0x02d627 to 0x01d097 (0x1058f bytes) 000198 1713531354.596968 mapwrite 0x011c41 thru 0x021b93 (0xff53 bytes) 000199 1713531354.625993 MIRROR_RESYNC 2 mirrors 000200 1713531354.701791 write 0x0335a6 thru 0x039fce (0x6a29 bytes) 000202 1713531354.715156 mapread 0x02a0b8 thru 0x02ee7b (0x4dc4 bytes) 000203 1713531354.719475 write 0x03c7b1 thru 0x03ffff (0x384f bytes) 000204 1713531354.720200 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000209 1713531354.749917 mapread 0x00fc1b thru 0x01e6fa (0xeae0 bytes) 000210 1713531354.756831 mapwrite 0x0071c6 thru 0x012c6a (0xbaa5 bytes) 000212 1713531354.775384 read 0x00a313 thru 0x011b30 (0x781e bytes) 000214 1713531354.775911 mapwrite 0x01ab52 thru 0x023012 (0x84c1 bytes) 000215 1713531354.790941 trunc from 0x040000 to 0x03272b (0xd8d4 bytes) 000217 1713531354.798322 write 0x01d36c thru 0x023b55 (0x67ea bytes) 000219 1713531354.799520 mapwrite 0x02ff06 thru 0x038009 (0x8104 bytes) 000222 1713531354.820012 write 0x029939 thru 0x0302eb (0x69b3 bytes) 000226 1713531354.821106 read 0x0280ef thru 0x0336b9 (0xb5cb bytes) 000227 1713531354.821389 mapwrite 0x02ca08 thru 0x038797 (0xbd90 bytes) 000228 1713531354.839318 trunc from 0x038798 to 0x01d92d (0x1ae6a bytes) 000229 1713531354.842936 trunc from 0x01d92e to 0x0204d8 (0x2bab bytes) 000231 1713531354.846679 write 0x039f69 thru 0x03b660 (0x16f8 bytes) 000233 1713531354.847372 mapread 0x023b55 thru 0x02c926 (0x8dd2 bytes) 000235 1713531354.850448 mapwrite 0x03f70f thru 0x03ffff (0x8f1 bytes) 000236 1713531354.869006 mapwrite 0x01a724 thru 0x01ed5b (0x4638 bytes) 000243 1713531354.882056 write 0x000029 thru 0x00c976 (0xc94e bytes) 000249 1713531354.884137 read 0x01b676 thru 0x024217 (0x8ba2 bytes) 000250 1713531354.885899 MIRROR_EXTEND to 2 mirrors 000251 1713531354.997754 write 0x014d8b thru 0x01ea62 (0x9cd8 bytes) 000252 1713531355.014392 MIRROR_EXTEND to 3 mirrors 000254 1713531355.125735 mapwrite 0x007b6c thru 0x00f178 (0x760d bytes) 000255 1713531355.169034 write 0x0053b3 thru 0x009e8d (0x4adb bytes) 000257 1713531355.170115 MIRROR_RESYNC 3 mirrors 000258 1713531355.283427 trunc from 0x040000 to 0x03ad70 (0x528f bytes) 000259 1713531355.297971 read 0x00743f thru 0x00a5ad (0x316f bytes) 000260 1713531355.299827 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000261 1713531355.337963 read 0x01034b thru 0x014282 (0x3f38 bytes) 000263 1713531355.340898 read 0x020d70 thru 0x02e1b0 (0xd441 bytes) 000265 1713531355.344183 mapread 0x00b7be thru 0x012e48 (0x768b bytes) 000269 1713531355.345410 MIRROR_RESYNC 2 mirrors 000271 1713531355.431399 mapread 0x02219d thru 0x022ac9 (0x92d bytes) 000274 1713531355.433203 MIRROR_RESYNC 2 mirrors 000275 1713531355.459050 mapwrite 0x007f8b thru 0x016563 (0xe5d9 bytes) 000276 1713531355.504560 mapread 0x02ab99 thru 0x032239 (0x76a1 bytes) 000277 1713531355.509070 read 0x00f676 thru 0x011e27 (0x27b2 bytes) 000282 1713531355.511028 MIRROR_EXTEND to 3 mirrors 000283 1713531355.616151 mapread 0x02cf6d thru 0x036863 (0x98f7 bytes) 000284 1713531355.624444 MIRROR_EXTEND to 4 mirrors 000285 1713531355.723264 mapread 0x0139a7 thru 0x01c201 (0x885b bytes) 000291 1713531355.728549 mapread 0x0353fb thru 0x03ad70 (0x5976 bytes) 000292 1713531355.730500 mapwrite 0x017952 thru 0x01bc63 (0x4312 bytes) 000293 1713531355.770233 MIRROR_RESYNC 4 mirrors 000295 1713531355.893242 trunc from 0x03ad71 to 0x00db6b (0x2d205 bytes) 000297 1713531355.912396 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000300 1713531355.953996 mapread 0x0047ac thru 0x00db6b (0x93c0 bytes) 000302 1713531355.959150 read 0x002044 thru 0x005fc7 (0x3f84 bytes) 000303 1713531355.960765 read 0x00086e thru 0x009f4a (0x96dd bytes) 000304 1713531355.962308 write 0x00febf thru 0x01061a (0x75c bytes) 000305 1713531355.963019 trunc from 0x01061b to 0x01f70e (0xf0f4 bytes) 000306 1713531355.966856 mapwrite 0x02a21d thru 0x02d116 (0x2efa bytes) 000307 1713531355.987434 mapread 0x0284f9 thru 0x02a92b (0x2433 bytes) 000308 1713531355.988051 write 0x02c42e thru 0x03287b (0x644e bytes) 000309 1713531355.988897 read 0x02ef62 thru 0x03287b (0x391a bytes) 000313 1713531355.990273 read 0x00c5e8 thru 0x01c1e3 (0xfbfc bytes) 000314 1713531355.990763 trunc from 0x03287c to 0x0134fd (0x1f37e bytes) 000315 1713531355.994631 write 0x016d77 thru 0x01b0ed (0x4377 bytes) 000316 1713531355.995334 MIRROR_EXTEND to 4 mirrors 000319 1713531356.107445 read 0x014985 thru 0x01b0ed (0x6769 bytes) 000322 1713531356.110051 mapread 0x0052b8 thru 0x007a3f (0x2788 bytes) 000324 1713531356.114305 mapread 0x011057 thru 0x01a76f (0x9719 bytes) 000325 1713531356.115807 mapread 0x005e60 thru 0x011cee (0xbe8f bytes) 000327 1713531356.116472 read 0x006bab thru 0x00a0d1 (0x3527 bytes) 000328 1713531356.116787 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000329 1713531356.170103 read 0x015a6d thru 0x01b0ed (0x5681 bytes) 000330 1713531356.175069 trunc from 0x01b0ee to 0x005cd6 (0x15417 bytes) truncating to largest ever: 0x3c9ab 000331 1713531356.191939 trunc from 0x005cd7 to 0x03c9aa (0x36cd4 bytes) 000334 1713531356.196982 write 0x020d60 thru 0x0211f8 (0x499 bytes) 000335 1713531356.199605 trunc from 0x03c9ab to 0x03b0b0 (0x18fa bytes) 000336 1713531356.203042 trunc from 0x03b0b1 to 0x037424 (0x3c8c bytes) 000340 1713531356.207660 read 0x0350db thru 0x037424 (0x234a bytes) 000341 1713531356.211071 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000342 1713531356.250683 read 0x010a07 thru 0x01a6c9 (0x9cc3 bytes) 000343 1713531356.253205 mapwrite 0x0378b4 thru 0x03daf9 (0x6246 bytes) 000344 1713531356.277291 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000345 1713531356.305840 mapread 0x005b70 thru 0x009d4c (0x41dd bytes) 000346 1713531356.309915 read 0x01ce17 thru 0x0277ff (0xa9e9 bytes) 000347 1713531356.312107 read 0x00bbd6 thru 0x0146b0 (0x8adb bytes) 000348 1713531356.314422 read 0x025515 thru 0x0288ed (0x33d9 bytes) 000349 1713531356.314731 write 0x026c6c thru 0x0314fd (0xa892 bytes) 000350 1713531356.315264 read 0x00578d thru 0x01300f (0xd883 bytes) 000351 1713531356.315582 write 0x02597c thru 0x034a3b (0xf0c0 bytes) 000352 1713531356.316210 mapread 0x038316 thru 0x03daf9 (0x57e4 bytes) 000354 1713531356.316825 trunc from 0x03dafa to 0x02840f (0x156ea bytes) 000356 1713531356.320745 write 0x035241 thru 0x03e6b1 (0x9471 bytes) 000358 1713531356.321905 read 0x0210b1 thru 0x0291c1 (0x8111 bytes) 000359 1713531356.323961 mapread 0x012814 thru 0x01f01c (0xc809 bytes) 000362 1713531356.324943 mapread 0x03c225 thru 0x03cc29 (0xa05 bytes) 000363 1713531356.325295 read 0x00bea3 thru 0x00bea9 (0x7 bytes) 000367 1713531356.326122 read 0x0290ec thru 0x02ae69 (0x1d7e bytes) 000369 1713531356.326578 read 0x0046d5 thru 0x0102c9 (0xbbf5 bytes) 000371 1713531356.327052 trunc from 0x03e6b2 to 0x02001a (0x1e697 bytes) 000373 1713531356.330907 write 0x02f6fa thru 0x039847 (0xa14e bytes) 000375 1713531356.331683 read 0x004ba2 thru 0x0145c1 (0xfa20 bytes) 000376 1713531356.331969 write 0x011799 thru 0x013af2 (0x235a bytes) 000377 1713531356.332324 write 0x01c17e thru 0x01df25 (0x1da8 bytes) 000378 1713531356.332681 read 0x01f8ed thru 0x02ec08 (0xf31c bytes) 000380 1713531356.335278 write 0x000b25 thru 0x001608 (0xae4 bytes) 000383 1713531356.336620 trunc from 0x039848 to 0x032035 (0x7812 bytes) 000387 1713531356.340539 mapwrite 0x0304af thru 0x03ad19 (0xa86b bytes) 000389 1713531356.366531 write 0x033617 thru 0x03ffff (0xc9e9 bytes) 000390 1713531356.367166 write 0x038d46 thru 0x03c4b4 (0x376f bytes) 000391 1713531356.367626 trunc from 0x040000 to 0x03ba45 (0x45ba bytes) 000392 1713531356.371053 mapread 0x01a708 thru 0x025048 (0xa941 bytes) 000393 1713531356.371521 mapread 0x004a87 thru 0x00f186 (0xa700 bytes) 000394 1713531356.371906 trunc from 0x03ba46 to 0x01d282 (0x1e7c3 bytes) 000395 1713531356.375067 mapwrite 0x03a5b0 thru 0x03ffff (0x5a50 bytes) 000397 1713531356.394930 write 0x025bf3 thru 0x02eaf0 (0x8efe bytes) 000399 1713531356.398254 write 0x02f4e6 thru 0x02f7a4 (0x2bf bytes) 000400 1713531356.399925 trunc from 0x040000 to 0x01bf21 (0x240de bytes) 000404 1713531356.404342 mapread 0x00b33f thru 0x010605 (0x52c7 bytes) 000408 1713531356.405650 trunc from 0x01bf22 to 0x0176aa (0x4877 bytes) 000410 1713531356.409127 mapwrite 0x010221 thru 0x01dd99 (0xdb79 bytes) 000413 1713531356.435579 mapread 0x00c092 thru 0x0112ef (0x525e bytes) 000414 1713531356.436042 trunc from 0x01dd9a to 0x011df1 (0xbfa8 bytes) 000415 1713531356.439914 read 0x00e66e thru 0x011df1 (0x3784 bytes) 000416 1713531356.440335 mapwrite 0x011211 thru 0x01817a (0x6f6a bytes) 000417 1713531356.459218 mapread 0x003cae thru 0x0068b8 (0x2c0b bytes) 000419 1713531356.459844 mapread 0x00ca8a thru 0x01817a (0xb6f1 bytes) 000425 1713531356.461280 mapread 0x00c1cc thru 0x017ba5 (0xb9da bytes) 000426 1713531356.461664 mapwrite 0x02437c thru 0x027694 (0x3319 bytes) 000427 1713531356.482338 read 0x01a8bc thru 0x0215e5 (0x6d2a bytes) 000428 1713531356.482716 trunc from 0x027695 to 0x039a3b (0x123a7 bytes) 000430 1713531356.486223 write 0x0151b8 thru 0x0160f2 (0xf3b bytes) 000431 1713531356.486643 mapread 0x03715f thru 0x039a3b (0x28dd bytes) 000435 1713531356.490141 trunc from 0x039a3c to 0x00ebe4 (0x2ae57 bytes) 000438 1713531356.494073 mapwrite 0x030f58 thru 0x039f22 (0x8fcb bytes) 000439 1713531356.522239 trunc from 0x039f23 to 0x007d1f (0x32203 bytes) 000441 1713531356.526036 write 0x00279f thru 0x0091d5 (0x6a37 bytes) 000443 1713531356.526780 write 0x03ad65 thru 0x03f275 (0x4511 bytes) 000444 1713531356.527276 write 0x03214b thru 0x03c406 (0xa2bc bytes) 000445 1713531356.529203 mapread 0x016934 thru 0x021b08 (0xb1d5 bytes) 000446 1713531356.542793 read 0x00047a thru 0x00e95f (0xe4e6 bytes) 000447 1713531356.545528 write 0x01d783 thru 0x02b896 (0xe114 bytes) 000448 1713531356.546527 read 0x034ea7 thru 0x035ff2 (0x114c bytes) 000449 1713531356.547024 trunc from 0x03f276 to 0x01c3fb (0x22e7a bytes) 000450 1713531356.551163 mapwrite 0x03654b thru 0x03f58a (0x9040 bytes) 000451 1713531356.574928 mapwrite 0x02d45a thru 0x02ea1c (0x15c3 bytes) 000452 1713531356.588151 read 0x00df37 thru 0x01dc9d (0xfd67 bytes) 000454 1713531356.589339 write 0x01036b thru 0x014dd4 (0x4a6a bytes) 000456 1713531356.590475 read 0x004349 thru 0x00e97b (0xa633 bytes) 000457 1713531356.590944 trunc from 0x03f58b to 0x0255eb (0x19f9f bytes) 000458 1713531356.595423 read 0x00fbfe thru 0x018211 (0x8614 bytes) 000460 1713531356.597018 MIRROR_EXTEND to 2 mirrors 000461 1713531356.696736 read 0x024888 thru 0x0255eb (0xd64 bytes) 000462 1713531356.698379 write 0x02e1fc thru 0x03c7bd (0xe5c2 bytes) 000463 1713531356.712308 read 0x013e37 thru 0x01aefc (0x70c6 bytes) 000465 1713531356.715433 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000470 1713531356.746080 read 0x0061b3 thru 0x007101 (0xf4f bytes) 000472 1713531356.748250 read 0x0333ff thru 0x03b196 (0x7d98 bytes) 000473 1713531356.751805 read 0x02e825 thru 0x0335f2 (0x4dce bytes) 000474 1713531356.752212 mapwrite 0x0219fd thru 0x022b7f (0x1183 bytes) 000476 1713531356.768483 mapread 0x03a43b thru 0x03c7bd (0x2383 bytes) 000477 1713531356.768887 read 0x013e78 thru 0x01f677 (0xb800 bytes) 000478 1713531356.769293 write 0x00376a thru 0x00b325 (0x7bbc bytes) 000479 1713531356.769978 mapread 0x01627e thru 0x018fce (0x2d51 bytes) 000480 1713531356.770319 trunc from 0x03c7be to 0x0369bd (0x5e00 bytes) 000481 1713531356.779841 MIRROR_EXTEND to 2 mirrors 000482 1713531356.888869 write 0x00ed4a thru 0x011a8c (0x2d43 bytes) truncating to largest ever: 0x3d1cd 000484 1713531356.905811 trunc from 0x0369be to 0x03d1cc (0x680f bytes) 000485 1713531356.912710 mapwrite 0x021d7c thru 0x0275f2 (0x5877 bytes) 000487 1713531356.930262 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000488 1713531356.957920 trunc from 0x03d1cd to 0x00a4cf (0x32cfd bytes) 000492 1713531356.962182 mapread 0x002cff thru 0x00a4cf (0x77d1 bytes) 000494 1713531356.967058 mapwrite 0x001c30 thru 0x00e845 (0xcc16 bytes) 000495 1713531356.987218 read 0x00d001 thru 0x00e845 (0x1845 bytes) 000496 1713531356.987598 write 0x0270d3 thru 0x02b26f (0x419d bytes) 000497 1713531356.988273 trunc from 0x02b270 to 0x022512 (0x8d5d bytes) 000499 1713531356.992407 mapread 0x00053f thru 0x0028c9 (0x238b bytes) 000500 1713531356.994576 read 0x00b6c8 thru 0x00f273 (0x3bac bytes) 000501 1713531356.994856 mapwrite 0x038009 thru 0x03ffff (0x7ff7 bytes) 000504 1713531357.015852 trunc from 0x040000 to 0x03a832 (0x57cd bytes) 000505 1713531357.019422 mapwrite 0x020cec thru 0x029220 (0x8535 bytes) 000506 1713531357.035624 read 0x02846d thru 0x037ca3 (0xf837 bytes) 000508 1713531357.036473 mapread 0x022f66 thru 0x028b72 (0x5c0d bytes) 000509 1713531357.037094 mapread 0x0346d7 thru 0x034ec0 (0x7ea bytes) 000512 1713531357.038454 mapwrite 0x03c2da thru 0x03ffff (0x3d26 bytes) 000515 1713531357.058072 read 0x00a41e thru 0x016b9e (0xc781 bytes) 000516 1713531357.058502 mapread 0x039d39 thru 0x03ffff (0x62c7 bytes) 000518 1713531357.059282 write 0x0215c2 thru 0x026b8d (0x55cc bytes) 000524 1713531357.060828 read 0x02acc1 thru 0x034dd8 (0xa118 bytes) 000525 1713531357.061079 trunc from 0x040000 to 0x02d9e9 (0x12616 bytes) 000529 1713531357.065346 trunc from 0x02d9ea to 0x0314b7 (0x3ace bytes) 000535 1713531357.069500 mapwrite 0x03ffeb thru 0x03ffff (0x15 bytes) 000538 1713531357.087558 mapread 0x007523 thru 0x00fca5 (0x8783 bytes) 000539 1713531357.087958 mapread 0x02f5fe thru 0x03ed85 (0xf788 bytes) truncating to largest ever: 0x3ea89 000540 1713531357.088369 trunc from 0x040000 to 0x03ea88 (0x1577 bytes) 000541 1713531357.091519 write 0x01dd33 thru 0x021daa (0x4078 bytes) 000542 1713531357.093068 MIRROR_EXTEND to 2 mirrors 000543 1713531357.197735 read 0x024ce5 thru 0x02e848 (0x9b64 bytes) 000544 1713531357.199940 read 0x003642 thru 0x011935 (0xe2f4 bytes) 000546 1713531357.202736 read 0x021831 thru 0x028031 (0x6801 bytes) 000547 1713531357.202945 mapread 0x02b332 thru 0x031c7e (0x694d bytes) 000548 1713531357.203263 write 0x03d643 thru 0x03e0ee (0xaac bytes) 000552 1713531357.221898 read 0x00cc49 thru 0x0111ae (0x4566 bytes) 000554 1713531357.222406 trunc from 0x03ea89 to 0x03c483 (0x2605 bytes) 000555 1713531357.227020 MIRROR_EXTEND to 3 mirrors 000557 1713531357.335627 mapwrite 0x02fd78 thru 0x032e12 (0x309b bytes) 000560 1713531357.382166 mapread 0x038ebc thru 0x03c483 (0x35c8 bytes) 000561 1713531357.383942 read 0x027150 thru 0x02f6c5 (0x8576 bytes) 000562 1713531357.385568 MIRROR_RESYNC 3 mirrors 000563 1713531357.489571 mapwrite 0x028e5c thru 0x037baa (0xed4f bytes) 000565 1713531357.540376 MIRROR_RESYNC 3 mirrors 000566 1713531357.638762 MIRROR_EXTEND to 4 mirrors 000568 1713531357.741627 trunc from 0x03c484 to 0x0118f8 (0x2ab8b bytes) 000569 1713531357.762179 read 0x01173e thru 0x0118f8 (0x1bb bytes) 000570 1713531357.763866 mapread 0x007029 thru 0x0118f8 (0xa8d0 bytes) 000571 1713531357.768279 trunc from 0x0118f9 to 0x00e1c5 (0x3733 bytes) 000573 1713531357.773508 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000578 1713531357.816844 write 0x02b8c9 thru 0x03ab9e (0xf2d6 bytes) 000580 1713531357.818403 mapwrite 0x00a9d5 thru 0x00add9 (0x405 bytes) 000581 1713531357.834756 MIRROR_RESYNC 3 mirrors 000582 1713531357.937390 mapread 0x006717 thru 0x0123ad (0xbc97 bytes) 000584 1713531357.943159 write 0x039cad thru 0x03ffff (0x6353 bytes) 000585 1713531357.958646 write 0x03e9de thru 0x03ffff (0x1622 bytes) 000586 1713531357.959054 mapwrite 0x01d304 thru 0x02c23f (0xef3c bytes) 000587 1713531357.978528 MIRROR_EXTEND to 4 mirrors 000588 1713531358.087176 mapwrite 0x029b97 thru 0x02bb3f (0x1fa9 bytes) 000590 1713531358.127496 mapwrite 0x024a2f thru 0x030a7d (0xc04f bytes) 000591 1713531358.140515 mapread 0x00771a thru 0x01193d (0xa224 bytes) 000592 1713531358.144599 mapwrite 0x0211d9 thru 0x029f69 (0x8d91 bytes) 000595 1713531358.158554 read 0x022370 thru 0x0288ea (0x657b bytes) 000596 1713531358.158858 mapwrite 0x0244a1 thru 0x02edda (0xa93a bytes) 000600 1713531358.174595 trunc from 0x040000 to 0x022dc4 (0x1d23b bytes) 000601 1713531358.179464 mapread 0x022407 thru 0x022dc4 (0x9be bytes) 000602 1713531358.180058 read 0x02139b thru 0x022dc4 (0x1a2a bytes) 000603 1713531358.180721 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000604 1713531358.221676 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000605 1713531358.257780 mapread 0x00c624 thru 0x00e00c (0x19e9 bytes) 000606 1713531358.262345 write 0x01ff2c thru 0x0213e6 (0x14bb bytes) 000608 1713531358.264976 trunc from 0x022dc5 to 0x007d7d (0x1b047 bytes) 000609 1713531358.268584 write 0x007eaf thru 0x010a1b (0x8b6d bytes) 000614 1713531358.269925 trunc from 0x010a1c to 0x00a137 (0x68e4 bytes) 000617 1713531358.273818 read 0x007a79 thru 0x009867 (0x1def bytes) 000618 1713531358.274091 mapread 0x000561 thru 0x0080b0 (0x7b50 bytes) 000620 1713531358.276973 read 0x0013da thru 0x00a137 (0x8d5e bytes) 000621 1713531358.277477 MIRROR_RESYNC 2 mirrors 000622 1713531358.357947 trunc from 0x00a138 to 0x0255ce (0x1b497 bytes) 000623 1713531358.375122 read 0x0064df thru 0x015fbe (0xfae0 bytes) 000624 1713531358.377432 read 0x00b91a thru 0x013110 (0x77f7 bytes) 000625 1713531358.377926 MIRROR_EXTEND to 3 mirrors 000627 1713531358.475820 mapread 0x022c76 thru 0x0255ce (0x2959 bytes) 000630 1713531358.482584 mapwrite 0x02c571 thru 0x035e72 (0x9902 bytes) 000631 1713531358.521481 read 0x019f87 thru 0x01d7fc (0x3876 bytes) 000632 1713531358.524079 mapread 0x021af8 thru 0x02fd14 (0xe21d bytes) 000634 1713531358.527190 write 0x016b88 thru 0x0208b2 (0x9d2b bytes) 000636 1713531358.529906 mapread 0x005297 thru 0x00b3ee (0x6158 bytes) 000639 1713531358.532931 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000640 1713531358.571930 MIRROR_EXTEND to 3 mirrors 000641 1713531358.676430 trunc from 0x035e73 to 0x02c10d (0x9d65 bytes) 000642 1713531358.695963 mapread 0x021da4 thru 0x0296ca (0x7927 bytes) 000645 1713531358.701028 mapwrite 0x037e95 thru 0x03c32f (0x449b bytes) 000646 1713531358.723894 read 0x0382b8 thru 0x038909 (0x652 bytes) 000647 1713531358.724181 write 0x0360e5 thru 0x039651 (0x356d bytes) 000649 1713531358.724876 trunc from 0x03c330 to 0x017e9c (0x24493 bytes) 000650 1713531358.728868 MIRROR_RESYNC 3 mirrors 000651 1713531358.828957 mapread 0x00a710 thru 0x017e9c (0xd78d bytes) 000652 1713531358.835284 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000653 1713531358.875345 write 0x031951 thru 0x0391af (0x785f bytes) 000654 1713531358.889866 mapread 0x029a75 thru 0x02e69e (0x4c2a bytes) 000656 1713531358.894399 read 0x02c931 thru 0x0391af (0xc87f bytes) 000657 1713531358.894872 mapread 0x02529b thru 0x0270a0 (0x1e06 bytes) 000660 1713531358.896601 MIRROR_RESYNC 2 mirrors 000661 1713531358.974859 trunc from 0x0391b0 to 0x023cd4 (0x154db bytes) 000663 1713531358.992897 read 0x01dbe4 thru 0x023341 (0x575e bytes) 000664 1713531358.995157 mapread 0x014ed0 thru 0x0154e9 (0x61a bytes) 000665 1713531358.997540 read 0x00c1c9 thru 0x01ab6c (0xe9a4 bytes) 000666 1713531359.000099 mapread 0x00de05 thru 0x013207 (0x5403 bytes) 000667 1713531359.000779 MIRROR_EXTEND to 3 mirrors 000668 1713531359.104136 mapread 0x02337a thru 0x023cd4 (0x95b bytes) 000669 1713531359.108159 trunc from 0x023cd5 to 0x017562 (0xc772 bytes) 000673 1713531359.126471 MIRROR_RESYNC 3 mirrors 000675 1713531359.235120 read 0x00cf82 thru 0x017562 (0xa5e1 bytes) 000677 1713531359.237608 MIRROR_RESYNC 3 mirrors 000678 1713531359.262626 mapwrite 0x036f5f thru 0x03ffff (0x90a1 bytes) 000679 1713531359.296603 write 0x02f95b thru 0x03b1f4 (0xb89a bytes) 000680 1713531359.299526 read 0x019576 thru 0x01cda8 (0x3833 bytes) 000681 1713531359.301146 trunc from 0x040000 to 0x02443b (0x1bbc4 bytes) 000685 1713531359.306602 trunc from 0x02443c to 0x03ab37 (0x166fc bytes) 000686 1713531359.310043 mapread 0x039d12 thru 0x03ab37 (0xe26 bytes) 000687 1713531359.312816 mapread 0x0163bb thru 0x022087 (0xbccd bytes) 000690 1713531359.317157 read 0x019eb5 thru 0x01a4c5 (0x611 bytes) 000691 1713531359.317442 write 0x00b20d thru 0x016a19 (0xb80d bytes) 000696 1713531359.321531 read 0x00667e thru 0x00f284 (0x8c07 bytes) 000697 1713531359.321895 write 0x03abca thru 0x03ffff (0x5436 bytes) 000700 1713531359.322855 write 0x00bab1 thru 0x019032 (0xd582 bytes) 000701 1713531359.323829 write 0x03dd91 thru 0x03f7c3 (0x1a33 bytes) 000705 1713531359.324990 trunc from 0x040000 to 0x022f4a (0x1d0b5 bytes) 000706 1713531359.329121 write 0x021ec2 thru 0x02df7f (0xc0be bytes) 000707 1713531359.329815 trunc from 0x02df80 to 0x009677 (0x24908 bytes) 000709 1713531359.333418 mapwrite 0x00baea thru 0x010aa4 (0x4fbb bytes) 000710 1713531359.353640 MIRROR_RESYNC 3 mirrors 000712 1713531359.451319 mapread 0x00abb5 thru 0x00efd0 (0x441c bytes) 000714 1713531359.455713 write 0x03c93f thru 0x03ffff (0x36c1 bytes) 000716 1713531359.468571 mapread 0x035224 thru 0x03ffff (0xaddc bytes) 000719 1713531359.471034 write 0x0331e0 thru 0x037a53 (0x4874 bytes) 000720 1713531359.473247 write 0x00f482 thru 0x012b8f (0x370e bytes) 000722 1713531359.475589 read 0x00bcf5 thru 0x012db0 (0x70bc bytes) 000724 1713531359.476048 mapread 0x01fdd3 thru 0x02a205 (0xa433 bytes) 000725 1713531359.479905 write 0x006b6a thru 0x0077a9 (0xc40 bytes) 000726 1713531359.481840 read 0x000430 thru 0x00f340 (0xef11 bytes) 000727 1713531359.483743 trunc from 0x040000 to 0x022b65 (0x1d49a bytes) 000730 1713531359.492155 read 0x00e9ca thru 0x019306 (0xa93d bytes) 000731 1713531359.492655 mapread 0x016dc8 thru 0x022b65 (0xbd9e bytes) 000733 1713531359.493373 mapwrite 0x03555b thru 0x03884a (0x32f0 bytes) 000735 1713531359.514696 MIRROR_RESYNC 3 mirrors 000736 1713531359.614547 read 0x009d7e thru 0x019cba (0xff3d bytes) 000737 1713531359.617331 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000740 1713531359.663435 MIRROR_EXTEND to 3 mirrors 000741 1713531359.761924 read 0x020768 thru 0x02833a (0x7bd3 bytes) 000743 1713531359.765894 trunc from 0x03884b to 0x00a580 (0x2e2ca bytes) 000745 1713531359.785334 mapwrite 0x010bf1 thru 0x020475 (0xf885 bytes) 000746 1713531359.815553 mapwrite 0x022aad thru 0x02c326 (0x987a bytes) 000747 1713531359.836697 mapwrite 0x01a5bc thru 0x0212d6 (0x6d1b bytes) 000748 1713531359.850553 read 0x01ac82 thru 0x023d40 (0x90bf bytes) 000752 1713531359.851595 mapread 0x013d81 thru 0x01409e (0x31e bytes) 000753 1713531359.852005 read 0x02102d thru 0x023702 (0x26d6 bytes) 000755 1713531359.852455 read 0x023369 thru 0x025a9a (0x2732 bytes) 000756 1713531359.852847 mapread 0x00d246 thru 0x010333 (0x30ee bytes) 000758 1713531359.853719 read 0x01c8b3 thru 0x02484e (0x7f9c bytes) 000760 1713531359.854461 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000763 1713531359.893244 write 0x02139c thru 0x02e24c (0xceb1 bytes) 000767 1713531359.896231 read 0x000099 thru 0x00a9f7 (0xa95f bytes) 000768 1713531359.899243 read 0x01f50b thru 0x025c16 (0x670c bytes) 000773 1713531359.903654 MIRROR_RESYNC 2 mirrors 000774 1713531359.995831 read 0x02c13f thru 0x02e24c (0x210e bytes) 000775 1713531359.997811 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000776 1713531360.029064 mapread 0x017dbb thru 0x022898 (0xaade bytes) 000778 1713531360.038688 mapread 0x020efe thru 0x021d38 (0xe3b bytes) 000780 1713531360.039576 mapread 0x004981 thru 0x012349 (0xd9c9 bytes) 000781 1713531360.046996 trunc from 0x02e24d to 0x0326b3 (0x4467 bytes) 000783 1713531360.061011 write 0x02bad9 thru 0x02c913 (0xe3b bytes) 000788 1713531360.065785 trunc from 0x0326b4 to 0x00a806 (0x27ead bytes) 000790 1713531360.070251 mapread 0x002564 thru 0x0059a1 (0x343e bytes) 000791 1713531360.073209 trunc from 0x00a807 to 0x03757c (0x2cd76 bytes) 000795 1713531360.077639 read 0x002a7d thru 0x012857 (0xfddb bytes) 000796 1713531360.080050 mapread 0x012608 thru 0x0211ad (0xeba6 bytes) 000797 1713531360.084167 mapread 0x00f246 thru 0x015cca (0x6a85 bytes) 000799 1713531360.084732 write 0x01be31 thru 0x01c9df (0xbaf bytes) 000803 1713531360.086502 trunc from 0x03757d to 0x022938 (0x14c44 bytes) 000804 1713531360.090574 mapwrite 0x039a97 thru 0x03ed1e (0x5288 bytes) 000805 1713531360.110612 read 0x02c5c4 thru 0x02cead (0x8ea bytes) 000806 1713531360.113032 trunc from 0x03ed1f to 0x01da02 (0x2131c bytes) 000808 1713531360.116607 read 0x01ca5d thru 0x01da02 (0xfa6 bytes) 000809 1713531360.116882 mapwrite 0x0012e9 thru 0x002a9d (0x17b5 bytes) 000811 1713531360.130321 MIRROR_EXTEND to 2 mirrors 000812 1713531360.230733 mapwrite 0x03da49 thru 0x03ffff (0x25b7 bytes) 000813 1713531360.264324 write 0x03f21d thru 0x03ff5f (0xd43 bytes) 000815 1713531360.264969 trunc from 0x040000 to 0x039276 (0x6d89 bytes) 000817 1713531360.269242 mapread 0x0318f2 thru 0x039276 (0x7985 bytes) 000823 1713531360.272957 trunc from 0x039277 to 0x03356e (0x5d08 bytes) 000826 1713531360.277953 read 0x02130f thru 0x029a22 (0x8714 bytes) 000831 1713531360.281933 MIRROR_EXTEND to 3 mirrors 000832 1713531360.381582 read 0x021331 thru 0x02c475 (0xb145 bytes) 000833 1713531360.384901 mapread 0x012531 thru 0x01d068 (0xab38 bytes) 000834 1713531360.393246 read 0x0109dc thru 0x01b2e6 (0xa90b bytes) 000835 1713531360.394461 mapwrite 0x00aeb7 thru 0x0160ae (0xb1f8 bytes) 000836 1713531360.463446 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000837 1713531360.528295 mapwrite 0x03cade thru 0x03ffff (0x3522 bytes) 000838 1713531360.576071 mapwrite 0x015d86 thru 0x018ab1 (0x2d2c bytes) 000840 1713531360.595574 trunc from 0x040000 to 0x03a379 (0x5c86 bytes) 000841 1713531360.600034 mapread 0x00e90a thru 0x015b04 (0x71fb bytes) 000842 1713531360.600987 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000844 1713531360.630203 mapwrite 0x007595 thru 0x00d3cc (0x5e38 bytes) 000848 1713531360.650595 mapwrite 0x032767 thru 0x03ffff (0xd899 bytes) 000850 1713531360.674092 mapread 0x039d7f thru 0x03ffff (0x6281 bytes) 000851 1713531360.674687 trunc from 0x040000 to 0x03e71a (0x18e5 bytes) 000852 1713531360.678499 write 0x00a2ac thru 0x00be0a (0x1b5f bytes) 000853 1713531360.678998 read 0x024980 thru 0x03484a (0xfecb bytes) 000854 1713531360.681143 mapwrite 0x005522 thru 0x009c82 (0x4761 bytes) 000857 1713531360.693941 read 0x01a683 thru 0x01b880 (0x11fe bytes) 000859 1713531360.696854 mapread 0x010dce thru 0x01a13d (0x9370 bytes) 000860 1713531360.697376 mapwrite 0x02024c thru 0x02cb16 (0xc8cb bytes) 000861 1713531360.713103 trunc from 0x03e71b to 0x028a26 (0x15cf4 bytes) 000862 1713531360.718088 MIRROR_EXTEND to 2 mirrors 000863 1713531360.803570 mapwrite 0x0381a7 thru 0x03ffff (0x7e59 bytes) 000864 1713531360.840273 mapread 0x00e747 thru 0x01940a (0xacc4 bytes) 000865 1713531360.845007 MIRROR_EXTEND to 3 mirrors 000866 1713531360.950563 write 0x03a122 thru 0x03fc74 (0x5b53 bytes) 000867 1713531360.965394 write 0x0088df thru 0x00d0c9 (0x47eb bytes) 000870 1713531360.968096 read 0x03e13e thru 0x03ffff (0x1ec2 bytes) 000871 1713531360.968377 write 0x00af68 thru 0x018bca (0xdc63 bytes) 000872 1713531360.970179 write 0x0121a2 thru 0x015000 (0x2e5f bytes) 000874 1713531360.970795 write 0x0112ec thru 0x01ff92 (0xeca7 bytes) 000875 1713531360.972500 mapwrite 0x031bc8 thru 0x03a034 (0x846d bytes) 000876 1713531360.989227 read 0x03c8b7 thru 0x03dd1d (0x1467 bytes) 000877 1713531360.989537 write 0x02a99d thru 0x030729 (0x5d8d bytes) 000878 1713531360.990040 mapread 0x03a2bc thru 0x03ffff (0x5d44 bytes) 000879 1713531360.990473 read 0x018c84 thru 0x019a5d (0xdda bytes) 000881 1713531360.991141 MIRROR_RESYNC 3 mirrors 000883 1713531361.090774 mapread 0x03b6be thru 0x03e603 (0x2f46 bytes) 000884 1713531361.096240 mapread 0x00f964 thru 0x018539 (0x8bd6 bytes) 000885 1713531361.102399 MIRROR_EXTEND to 4 mirrors 000886 1713531361.213330 MIRROR_RESYNC 4 mirrors 000888 1713531361.246435 read 0x03d8d5 thru 0x03da40 (0x16c bytes) 000889 1713531361.247758 write 0x00d12d thru 0x01c61a (0xf4ee bytes) 000891 1713531361.264109 MIRROR_RESYNC 4 mirrors 000892 1713531361.384076 trunc from 0x040000 to 0x03528c (0xad73 bytes) 000893 1713531361.398771 trunc from 0x03528d to 0x02866e (0xcc1e bytes) 000894 1713531361.402747 write 0x023677 thru 0x02e1b9 (0xab43 bytes) 000899 1713531361.405214 trunc from 0x02e1ba to 0x031b24 (0x396b bytes) 000900 1713531361.408433 write 0x038bd1 thru 0x03f910 (0x6d40 bytes) 000901 1713531361.408970 write 0x029365 thru 0x03281c (0x94b8 bytes) 000902 1713531361.410705 trunc from 0x03f911 to 0x0197c1 (0x2614f bytes) 000903 1713531361.413952 MIRROR_RESYNC 4 mirrors 000905 1713531361.536099 MIRROR_EXTEND to 5 mirrors 000906 1713531361.637235 mapread 0x013171 thru 0x0197c1 (0x6651 bytes) 000908 1713531361.641692 trunc from 0x0197c2 to 0x00cee5 (0xc8dc bytes) 000910 1713531361.657892 write 0x03447d thru 0x036c37 (0x27bb bytes) 000911 1713531361.659014 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000912 1713531361.698247 mapread 0x00bbb8 thru 0x01674b (0xab94 bytes) 000914 1713531361.705008 mapwrite 0x02aca5 thru 0x0311ef (0x654b bytes) 000915 1713531361.721814 write 0x024f61 thru 0x0348bf (0xf95f bytes) 000916 1713531361.723750 read 0x024433 thru 0x028fcb (0x4b99 bytes) 000917 1713531361.724040 mapwrite 0x0074ea thru 0x013517 (0xc02e bytes) 000920 1713531361.738232 write 0x006cd2 thru 0x015e9f (0xf1ce bytes) 000921 1713531361.738892 mapwrite 0x005157 thru 0x01412e (0xefd8 bytes) 000922 1713531361.753670 read 0x00be51 thru 0x00f967 (0x3b17 bytes) 000924 1713531361.754142 read 0x02471e thru 0x03349a (0xed7d bytes) 000927 1713531361.754864 mapwrite 0x009c40 thru 0x010be1 (0x6fa2 bytes) 000929 1713531361.768969 trunc from 0x036c38 to 0x00bc35 (0x2b002 bytes) 000930 1713531361.772957 mapwrite 0x0010c6 thru 0x006dbe (0x5cf9 bytes) 000932 1713531361.788965 read 0x003e7f thru 0x00bc35 (0x7db7 bytes) 000933 1713531361.789373 mapwrite 0x02af2e thru 0x031dd2 (0x6ea5 bytes) 000935 1713531361.809544 trunc from 0x031dd3 to 0x038469 (0x6697 bytes) 000936 1713531361.813125 trunc from 0x03846a to 0x01a2dc (0x1e18d bytes) 000937 1713531361.817320 write 0x036edd thru 0x03ea6c (0x7b90 bytes) 000938 1713531361.818394 write 0x023175 thru 0x0329e1 (0xf86d bytes) 000939 1713531361.821865 write 0x029c7a thru 0x031fd8 (0x835f bytes) 000940 1713531361.822770 mapread 0x01ec92 thru 0x02c6d3 (0xda42 bytes) 000941 1713531361.831436 mapread 0x00e15d thru 0x00f161 (0x1005 bytes) 000946 1713531361.833107 mapread 0x0377d9 thru 0x038f8c (0x17b4 bytes) 000947 1713531361.833767 MIRROR_EXTEND to 5 mirrors 000952 1713531361.950902 mapread 0x028f78 thru 0x037e7e (0xef07 bytes) 000954 1713531361.956864 trunc from 0x03ea6d to 0x02f8d8 (0xf194 bytes) 000955 1713531361.976178 read 0x00b666 thru 0x015495 (0x9e30 bytes) 000956 1713531361.978641 read 0x00f46f thru 0x013e16 (0x49a8 bytes) 000957 1713531361.978976 write 0x011f5c thru 0x01a8d9 (0x897e bytes) 000965 1713531361.983324 mapread 0x0062e5 thru 0x00bce7 (0x5a03 bytes) 000966 1713531361.983744 read 0x0194b9 thru 0x01fa41 (0x6589 bytes) 000967 1713531361.985979 mapread 0x023a5e thru 0x02f8d8 (0xbe7b bytes) 000969 1713531361.986579 write 0x030acc thru 0x030e32 (0x367 bytes) 000972 1713531361.987314 mapwrite 0x02d395 thru 0x0356e5 (0x8351 bytes) 000976 1713531362.010255 mapread 0x017881 thru 0x024eb9 (0xd639 bytes) 000977 1713531362.010894 write 0x010c95 thru 0x012ba8 (0x1f14 bytes) 000979 1713531362.011744 MIRROR_RESYNC 5 mirrors 000981 1713531362.156376 write 0x00f014 thru 0x01c939 (0xd926 bytes) 000982 1713531362.172151 read 0x02c083 thru 0x0337e2 (0x7760 bytes) 000983 1713531362.174187 read 0x018df6 thru 0x026bb2 (0xddbd bytes) 000984 1713531362.176721 trunc from 0x0356e6 to 0x025dc0 (0xf925 bytes) 000985 1713531362.184526 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000987 1713531362.227281 write 0x02a112 thru 0x039268 (0xf157 bytes) 000988 1713531362.228430 write 0x0029cc thru 0x00a900 (0x7f35 bytes) 000989 1713531362.231030 mapread 0x01ab01 thru 0x02865e (0xdb5e bytes) 000990 1713531362.235603 mapread 0x003794 thru 0x011144 (0xd9b1 bytes) 000993 1713531362.238712 trunc from 0x039269 to 0x02e497 (0xadd1 bytes) 000994 1713531362.242591 mapread 0x02b78e thru 0x02e497 (0x2d0a bytes) 000996 1713531362.243241 write 0x0010fc thru 0x00c130 (0xb035 bytes) 000999 1713531362.245161 write 0x03e00e thru 0x03ffff (0x1ff2 bytes) All operations completed A-OK! PASS 70a (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 08:56:05 (1713531365) Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre2 Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5287296 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6688096 .. done Extending file size to 7182336 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7347680 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7404576 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7063968 .. Extending file size to 8085120 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7407520 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8144672 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8584480 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8706080 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9287008 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7696960 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8766272 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8853184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9029824 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 6361 6362 6363 6365 6368 done 192.168.202.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg238-client.virtnet /mnt/lustre2 (opts:) 192.168.202.138@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg238-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:57:11 (1713531431) Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre2 Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8463392 .. Extending file size to 5321600 .. failed Extending file size to 8257856 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8303712 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8657600 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8641312 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9196480 .. Extending file size to 9168672 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 9825 9826 9827 9829 9831 192.168.202.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg238-client.virtnet /mnt/lustre2 (opts:) 192.168.202.138@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg238-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:58:16 (1713531496) Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0312343 s, 67.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0716583 s, 29.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg238-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:58:23 (1713531503) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0245834 s, 85.3 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 08:58:26 (1713531506) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd90:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xd8a:0x0] } PASS 203 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:58:29 (1713531509) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0226189 s, 92.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd91:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd8b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd8c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:58:32 (1713531512) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0329617 s, 63.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd93:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd8d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd93:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd8d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd8e:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:58:36 (1713531516) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0239418 s, 87.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd8f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd8f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd90:0x0] } PASS 204c (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:59:00 (1713531540) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg238-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0258758 s, 40.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd95:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd95:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd96:0x0] } PASS 204d (43s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:59:44 (1713531584) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg238-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0209697 s, 100 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd94:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd97:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd94:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd97:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:00:29 (1713531629) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg238-server mds-ost sync done. Creating new pool oleg238-server: Pool lustre.test_204f created Adding targets to pool oleg238-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg238-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0227903 s, 92.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd98:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd98:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg238-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg238-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg238-server: Pool lustre.test_204f destroyed PASS 204f (50s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer flag ========================================================== 09:01:21 (1713531681) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:01:25 (1713531685) oleg238-server: Pool lustre.test_206 created Adding targets to pool oleg238-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg238-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg238-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg238-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg238-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:01:42 (1713531702) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332446 s, 31.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133683 s, 78.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xf1841933 Mirror 3: 0xf1841933 PASS 207 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:01:47 (1713531707) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:01:51 (1713531711) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:01:54 (1713531714) Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0308504 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x25b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x770000 Read this (4 bytes): BBBB 192.168.202.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg238-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:02:13 (1713531733) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00361415 s, 2.3 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 09:02:17 (1713531737) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00881118 s, 119 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xadb lmm_fid: [0x200000404:0xadb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xadb lmm_fid: [0x200000404:0xadb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd9d:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836699 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-19 09:02:18.000000000 -0400 Modify: 2024-04-19 09:02:18.000000000 -0400 Change: 2024-04-19 09:02:18.000000000 -0400 Birth: - PASS 210a (123s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 885 sec ============ 09:04:21 (1713531861) === sanity-flr: start cleanup 09:04:21 (1713531861) === === sanity-flr: finish cleanup 09:04:22 (1713531862) ===