-----============= acceptance-small: sanity-flr ============----- Tue Apr 16 15:40:29 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:40:33 (1713296433) === oleg359-client.virtnet: executing check_config_client /mnt/lustre oleg359-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg359-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012e269800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012e269800.idle_timeout=debug disable quota as required oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:40:38 (1713296438) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6897] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 15:40:39 (1713296439) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:40:44 (1713296444) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:40:46 (1713296446) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 15:40:49 (1713296449) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:40:55 (1713296455) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:40:57 (1713296457) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 15:40:59 (1713296459) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012e269800.stats=clear osc.lustre-OST0001-osc-ffff88012e269800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:41:02 (1713296462) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:41:06 (1713296466) PASS 0j (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:41:09 (1713296469) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 15:41:11 (1713296471) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 15:41:15 (1713296475) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:41:18 (1713296478) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:41:20 (1713296480) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:41:23 (1713296483) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 15:41:27 (1713296487) PASS 7 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:41:29 (1713296489) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.317487 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0340298 s, 30.8 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:41:33 (1713296493) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0181324 s, 173 MB/s PASS 22 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:41:36 (1713296496) Stopping /mnt/lustre-ost1 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e269800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e269800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg359-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e269800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e269800.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg359-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:41:57 (1713296517) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.107002 s, 68.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e269800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e269800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg359-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:42:06 (1713296526) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e269800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e269800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg359-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e269800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e269800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg359-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:42:22 (1713296542) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.757057 s, 69.3 MB/s -rw-r--r-- 1 root root 52428800 Apr 16 15:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.195s user 0m0.002s sys 0m0.110s Stopping /mnt/lustre-ost1 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e269800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e269800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.193s user 0m0.000s sys 0m0.120s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg359-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e269800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e269800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.190s user 0m0.001s sys 0m0.110s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg359-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012e269800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e269800.ost_server_uuid in FULL state after 0 sec PASS 33b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:42:43 (1713296563) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:42:45 (1713296565) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:42:48 (1713296568) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 15:42:50 (1713296570) PASS 35 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 15:42:53 (1713296573) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.118072 s, 107 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:42:57 (1713296577) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104186 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101963 s, 103 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:43:00 (1713296580) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:43:04 (1713296584) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000654267 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000551747 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000576552 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000522301 s, 2.0 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 15:43:08 (1713296588) create test files with size 15751331 .. 1a53bc943452d6ab956f3d46e1a34e7e - e3eee4bd20318babfa5bd820f76b8c88 - 85b44df8313cea2e78fd77669c61c6b3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg359-client.virtnet /mnt/lustre (opts:) Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 15:43:14 (1713296594) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:43:18 (1713296598) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:43:21 (1713296601) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244301 s, 42.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x49:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00890764 s, 118 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 15:43:24 (1713296604) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0249802 s, 84.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0791281 s, 53.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0377987 s, 55.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 15:43:28 (1713296608) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 15:43:31 (1713296611) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:43:33 (1713296613) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00744236 s, 141 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 15:43:36 (1713296616) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:43:39 (1713296619) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:43:42 (1713296622) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.056481 s, 186 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg359-client.virtnet /mnt/lustre (opts:) Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:43:46 (1713296626) llite.lustre-ffff8800a8132800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0650937 s, 161 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg359-client.virtnet /mnt/lustre (opts:) Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff880130dec800.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:43:51 (1713296631) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 15:43:59 (1713296639) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 15:44:05 (1713296645) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 15:44:08 (1713296648) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0626978 s, 50.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0435327 s, 72.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 808cd53ffdcfbb0db8a6981557d6e3f3 - mirror 1 checksum: 550de2dda2a440f70ff552f4c11bc3a0 - mirror 2 checksum: 808cd53ffdcfbb0db8a6981557d6e3f3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 15:44:08 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 16 15:44:08 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 808cd53ffdcfbb0db8a6981557d6e3f3 - mirror 1 checksum: 550de2dda2a440f70ff552f4c11bc3a0 - mirror 2 checksum: 550de2dda2a440f70ff552f4c11bc3a0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 15:44:08 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 16 15:44:08 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:44:12 (1713296652) Failing mds1 on oleg359-server Stopping /mnt/lustre-mds1 (opts:) on oleg359-server 15:44:13 (1713296653) shut down Failover mds1 to oleg359-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 15:44:26 (1713296666) targets are mounted 15:44:26 (1713296666) facet_failover done oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00879668 s, 119 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 50A (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:44:32 (1713296672) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 15:44:35 (1713296675) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:44:37 (1713296677) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00185846 s, 8.8 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 15:44:40 (1713296680) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:44:43 (1713296683) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:44:45 (1713296685) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:44:50 (1713296690) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:45:09 (1713296709) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:45:27 (1713296727) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:45:45 (1713296745) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 15:45:48 (1713296748) mirror create pid 26594 mirror split pid 26595 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26594 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26595 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 15:46:51 (1713296811) Seed set to 6811 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1713296811.613969 write 0x026c05 thru 0x02a379 (0x3775 bytes) 000002 1713296811.616344 read 0x002bf9 thru 0x0077f5 (0x4bfd bytes) 000003 1713296811.618150 mapread 0x017aeb thru 0x01eab4 (0x6fca bytes) 000006 1713296811.622659 read 0x012f2b thru 0x01358c (0x662 bytes) 000009 1713296811.625172 write 0x001ad6 thru 0x00872b (0x6c56 bytes) 000012 1713296811.628404 mapwrite 0x03fbb1 thru 0x03ffff (0x44f bytes) 000013 1713296811.654645 mapread 0x0001b5 thru 0x00104e (0xe9a bytes) truncating to largest ever: 0x1277b 000014 1713296811.655120 trunc from 0x040000 to 0x01277a (0x2d885 bytes) 000015 1713296811.658512 trunc from 0x01277b to 0x00ea1c (0x3d5e bytes) 000017 1713296811.661679 read 0x003a79 thru 0x00ea1c (0xafa4 bytes) 000018 1713296811.662019 write 0x016907 thru 0x01bb11 (0x520b bytes) 000019 1713296811.662595 mapwrite 0x0016dd thru 0x011323 (0xfc47 bytes) 000020 1713296811.679334 mapwrite 0x0181d8 thru 0x0198ac (0x16d5 bytes) 000021 1713296811.690239 MIRROR_EXTEND to 2 mirrors 000022 1713296811.770944 mapwrite 0x009a7b thru 0x0112c2 (0x7848 bytes) 000024 1713296811.812459 mapread 0x00d194 thru 0x014d86 (0x7bf3 bytes) truncating to largest ever: 0x187e8 000025 1713296811.812964 trunc from 0x01bb12 to 0x0187e7 (0x332a bytes) 000029 1713296811.820507 write 0x0318ea thru 0x0331d9 (0x18f0 bytes) 000032 1713296811.821684 trunc from 0x0331da to 0x00f374 (0x23e65 bytes) 000035 1713296811.825195 mapwrite 0x00e83e thru 0x014773 (0x5f36 bytes) 000036 1713296811.844211 mapwrite 0x025931 thru 0x02e3bf (0x8a8f bytes) 000037 1713296811.863600 read 0x0114fe thru 0x016206 (0x4d09 bytes) 000038 1713296811.865048 mapwrite 0x02151a thru 0x028961 (0x7448 bytes) 000039 1713296811.876767 mapread 0x00c70b thru 0x014dbf (0x86b5 bytes) 000040 1713296811.877177 write 0x02999d thru 0x032be0 (0x9244 bytes) 000041 1713296811.878932 MIRROR_EXTEND to 3 mirrors 000042 1713296811.983294 write 0x03ec7d thru 0x03ffff (0x1383 bytes) truncating to largest ever: 0x3c7dc 000043 1713296811.995192 trunc from 0x040000 to 0x03c7db (0x3824 bytes) 000045 1713296812.002783 MIRROR_RESYNC 3 mirrors 000046 1713296812.099197 mapread 0x006c4f thru 0x00dc4e (0x7000 bytes) 000047 1713296812.102997 mapwrite 0x032da0 thru 0x0331cb (0x42c bytes) 000048 1713296812.142340 MIRROR_EXTEND to 4 mirrors 000050 1713296812.242152 trunc from 0x03c7dc to 0x033adb (0x8d00 bytes) 000051 1713296812.256865 MIRROR_RESYNC 4 mirrors 000052 1713296812.376376 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000054 1713296812.423317 trunc from 0x033adc to 0x011fc6 (0x21b15 bytes) 000056 1713296812.436636 read 0x008dcc thru 0x011fc6 (0x91fb bytes) 000057 1713296812.438759 read 0x003209 thru 0x0094cb (0x62c3 bytes) 000058 1713296812.440840 mapwrite 0x0367dc thru 0x03ffff (0x9824 bytes) 000059 1713296812.462957 trunc from 0x040000 to 0x009eaf (0x36150 bytes) 000061 1713296812.466600 mapwrite 0x0393df thru 0x03bb4d (0x276f bytes) 000062 1713296812.484409 mapwrite 0x0304d9 thru 0x036380 (0x5ea8 bytes) 000064 1713296812.502508 trunc from 0x03bb4e to 0x01b67a (0x204d3 bytes) 000066 1713296812.506813 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000067 1713296812.543839 mapread 0x00e43d thru 0x0166ad (0x8271 bytes) 000068 1713296812.548061 trunc from 0x01b67b to 0x016439 (0x5241 bytes) 000069 1713296812.551524 write 0x01d61e thru 0x026edd (0x98c0 bytes) 000070 1713296812.553637 MIRROR_EXTEND to 3 mirrors 000071 1713296812.653031 mapwrite 0x0297b0 thru 0x035340 (0xbb91 bytes) 000072 1713296812.689282 MIRROR_RESYNC 3 mirrors 000073 1713296812.788564 MIRROR_EXTEND to 4 mirrors 000075 1713296812.883804 trunc from 0x035341 to 0x0223c0 (0x12f80 bytes) 000076 1713296812.903240 trunc from 0x0223c1 to 0x0275c5 (0x5205 bytes) 000078 1713296812.908228 trunc from 0x0275c6 to 0x007a8a (0x1fb3b bytes) 000079 1713296812.911199 mapwrite 0x027ddc thru 0x02c321 (0x4546 bytes) 000081 1713296812.933194 MIRROR_RESYNC 4 mirrors 000082 1713296813.049445 read 0x025623 thru 0x02c321 (0x6cff bytes) 000086 1713296813.051763 write 0x032baf thru 0x03414b (0x159d bytes) 000087 1713296813.063164 mapread 0x01b460 thru 0x025a6f (0xa610 bytes) 000088 1713296813.069033 mapread 0x02d88c thru 0x03414b (0x68c0 bytes) 000089 1713296813.072068 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000090 1713296813.111727 mapread 0x00c3c2 thru 0x00fc56 (0x3895 bytes) 000092 1713296813.115844 mapread 0x02f8ff thru 0x0337ec (0x3eee bytes) 000095 1713296813.119644 MIRROR_EXTEND to 4 mirrors 000097 1713296813.220103 trunc from 0x03414c to 0x009275 (0x2aed6 bytes) 000098 1713296813.234085 read 0x000ef5 thru 0x0061b6 (0x52c2 bytes) 000100 1713296813.236323 read 0x00557f thru 0x009275 (0x3cf7 bytes) 000102 1713296813.238177 read 0x006934 thru 0x006a0c (0xd9 bytes) 000103 1713296813.238571 read 0x00417d thru 0x009275 (0x50f9 bytes) 000108 1713296813.240219 read 0x001320 thru 0x009275 (0x7f56 bytes) 000114 1713296813.242271 mapread 0x0017f1 thru 0x009275 (0x7a85 bytes) 000115 1713296813.242930 mapwrite 0x012734 thru 0x013d00 (0x15cd bytes) 000116 1713296813.260834 mapwrite 0x031f5f thru 0x03bb34 (0x9bd6 bytes) 000117 1713296813.280256 write 0x037e4e thru 0x03ffff (0x81b2 bytes) 000118 1713296813.280901 trunc from 0x040000 to 0x001ee4 (0x3e11b bytes) 000119 1713296813.284815 trunc from 0x001ee5 to 0x012a85 (0x10ba1 bytes) 000123 1713296813.288822 mapread 0x003c52 thru 0x00f2be (0xb66d bytes) 000125 1713296813.301528 MIRROR_RESYNC 4 mirrors 000126 1713296813.404516 write 0x0392da thru 0x03ffff (0x6d26 bytes) 000128 1713296813.418156 trunc from 0x040000 to 0x0170cf (0x28f30 bytes) 000129 1713296813.425621 MIRROR_RESYNC 4 mirrors 000130 1713296813.537500 trunc from 0x0170d0 to 0x0177e5 (0x716 bytes) 000131 1713296813.553877 write 0x03566e thru 0x03ffff (0xa992 bytes) 000134 1713296813.556289 MIRROR_RESYNC 4 mirrors 000135 1713296813.661998 write 0x032531 thru 0x034192 (0x1c62 bytes) 000136 1713296813.674521 trunc from 0x040000 to 0x006fc5 (0x3903a bytes) 000137 1713296813.680397 read 0x004307 thru 0x006fc5 (0x2cbf bytes) 000138 1713296813.681735 trunc from 0x006fc6 to 0x024fc5 (0x1e000 bytes) 000139 1713296813.685226 MIRROR_EXTEND to 5 mirrors 000141 1713296813.773681 trunc from 0x024fc6 to 0x03ace8 (0x15d23 bytes) 000143 1713296813.789562 write 0x00d8c0 thru 0x017eb3 (0xa5f4 bytes) 000144 1713296813.791900 mapread 0x011d33 thru 0x0209f7 (0xecc5 bytes) 000146 1713296813.795227 mapwrite 0x0153a7 thru 0x01dfdf (0x8c39 bytes) 000147 1713296813.807752 read 0x002363 thru 0x002ed1 (0xb6f bytes) 000148 1713296813.809203 mapread 0x0324d8 thru 0x0330ad (0xbd6 bytes) 000149 1713296813.811246 write 0x03b44a thru 0x03ffff (0x4bb6 bytes) 000150 1713296813.812898 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000151 1713296813.852453 write 0x007b07 thru 0x00fdbc (0x82b6 bytes) 000152 1713296813.854920 read 0x01a465 thru 0x01c9d0 (0x256c bytes) 000153 1713296813.856347 read 0x03b608 thru 0x03ffff (0x49f8 bytes) 000155 1713296813.859336 read 0x00c535 thru 0x010079 (0x3b45 bytes) 000157 1713296813.861161 MIRROR_EXTEND to 5 mirrors 000160 1713296813.968600 write 0x02b102 thru 0x02e141 (0x3040 bytes) 000161 1713296813.984285 write 0x02a03e thru 0x038bcf (0xeb92 bytes) 000162 1713296813.987803 mapwrite 0x034e40 thru 0x03ffff (0xb1c0 bytes) 000163 1713296814.004454 write 0x0109ba thru 0x0199b2 (0x8ff9 bytes) 000164 1713296814.007253 mapwrite 0x036752 thru 0x03e061 (0x7910 bytes) 000166 1713296814.019580 write 0x016a09 thru 0x01743c (0xa34 bytes) 000167 1713296814.020018 mapwrite 0x0085f3 thru 0x017081 (0xea8f bytes) 000168 1713296814.033935 mapwrite 0x0056a1 thru 0x0066ef (0x104f bytes) 000169 1713296814.047791 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000173 1713296814.089649 mapwrite 0x03a312 thru 0x03ffff (0x5cee bytes) 000174 1713296814.108104 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000175 1713296814.147406 trunc from 0x040000 to 0x01eb69 (0x21496 bytes) 000178 1713296814.154131 mapwrite 0x022647 thru 0x029407 (0x6dc1 bytes) 000180 1713296814.176057 write 0x0005ef thru 0x008839 (0x824b bytes) 000181 1713296814.180645 mapwrite 0x009bc4 thru 0x013d44 (0xa181 bytes) 000183 1713296814.195151 mapread 0x00a895 thru 0x014910 (0xa07c bytes) 000186 1713296814.196355 trunc from 0x029408 to 0x00e2b2 (0x1b155 bytes) 000189 1713296814.201428 MIRROR_RESYNC 3 mirrors 000190 1713296814.300938 trunc from 0x00e2b3 to 0x00cd11 (0x15a1 bytes) 000193 1713296814.314275 mapread 0x00307b thru 0x00ad8c (0x7d12 bytes) 000194 1713296814.318329 write 0x0235ee thru 0x0329ea (0xf3fd bytes) 000197 1713296814.321407 MIRROR_EXTEND to 4 mirrors 000198 1713296814.409124 trunc from 0x0329eb to 0x0081ec (0x2a7fe bytes) 000199 1713296814.426253 read 0x002184 thru 0x0081ec (0x6069 bytes) 000201 1713296814.428476 trunc from 0x0081ed to 0x015863 (0xd677 bytes) 000202 1713296814.433480 MIRROR_RESYNC 4 mirrors 000203 1713296814.539563 mapwrite 0x01ee2f thru 0x02cadb (0xdcad bytes) 000204 1713296814.574965 mapwrite 0x013cf2 thru 0x015316 (0x1625 bytes) 000205 1713296814.587912 write 0x0094bc thru 0x009b11 (0x656 bytes) 000206 1713296814.590102 read 0x0135c6 thru 0x01d799 (0xa1d4 bytes) 000207 1713296814.591558 MIRROR_EXTEND to 5 mirrors 000208 1713296814.680829 read 0x0169db thru 0x0245ca (0xdbf0 bytes) 000209 1713296814.683742 write 0x00610a thru 0x00d4e1 (0x73d8 bytes) 000210 1713296814.697750 read 0x001dc4 thru 0x009706 (0x7943 bytes) 000211 1713296814.700198 mapread 0x008d6f thru 0x0169f2 (0xdc84 bytes) 000212 1713296814.700717 write 0x00f1a9 thru 0x011768 (0x25c0 bytes) 000213 1713296814.701225 mapread 0x01ddae thru 0x01e43f (0x692 bytes) 000214 1713296814.701606 write 0x036201 thru 0x03ffff (0x9dff bytes) 000215 1713296814.702258 mapwrite 0x025131 thru 0x02b67c (0x654c bytes) 000216 1713296814.716578 MIRROR_EXTEND to 6 mirrors 000218 1713296814.808387 trunc from 0x040000 to 0x031fc4 (0xe03b bytes) 000219 1713296814.826492 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000222 1713296814.867516 trunc from 0x031fc5 to 0x037ed8 (0x5f14 bytes) 000224 1713296814.872381 MIRROR_RESYNC 5 mirrors 000225 1713296815.011266 read 0x01e491 thru 0x02c162 (0xdcd2 bytes) 000226 1713296815.013232 write 0x034e69 thru 0x03fc93 (0xae2b bytes) 000228 1713296815.026730 read 0x00a19d thru 0x01784e (0xd6b2 bytes) 000230 1713296815.029796 write 0x01db8d thru 0x023b1f (0x5f93 bytes) 000232 1713296815.030595 mapwrite 0x011fb5 thru 0x0127cb (0x817 bytes) 000233 1713296815.042743 write 0x0123a4 thru 0x01d91b (0xb578 bytes) 000234 1713296815.044326 MIRROR_EXTEND to 6 mirrors 000241 1713296815.155109 mapwrite 0x020db8 thru 0x02c587 (0xb7d0 bytes) 000242 1713296815.199532 write 0x006fc8 thru 0x00c1f5 (0x522e bytes) 000244 1713296815.202165 trunc from 0x03fc94 to 0x0295b3 (0x166e0 bytes) 000246 1713296815.209839 trunc from 0x0295b4 to 0x026e97 (0x271c bytes) 000248 1713296815.213485 write 0x0153c2 thru 0x025224 (0xfe63 bytes) 000250 1713296815.217664 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000251 1713296815.259330 mapread 0x004c9a thru 0x009c03 (0x4f6a bytes) 000252 1713296815.267905 MIRROR_EXTEND to 6 mirrors 000253 1713296815.365143 MIRROR_RESYNC 6 mirrors 000254 1713296815.497458 trunc from 0x026e98 to 0x032a64 (0xbbcd bytes) 000258 1713296815.514231 read 0x030b42 thru 0x032a64 (0x1f23 bytes) 000259 1713296815.515717 write 0x022cf7 thru 0x02638c (0x3696 bytes) 000261 1713296815.519117 read 0x00a86b thru 0x0149ff (0xa195 bytes) 000262 1713296815.521836 write 0x03ae46 thru 0x03ffff (0x51ba bytes) 000263 1713296815.522334 read 0x005b04 thru 0x013fe5 (0xe4e2 bytes) 000264 1713296815.522606 read 0x022ea8 thru 0x02d51d (0xa676 bytes) 000265 1713296815.522841 mapwrite 0x0323c1 thru 0x038e5e (0x6a9e bytes) 000266 1713296815.540025 mapwrite 0x006853 thru 0x00bed5 (0x5683 bytes) 000267 1713296815.552877 mapread 0x00db30 thru 0x01bbe0 (0xe0b1 bytes) 000268 1713296815.553301 read 0x020961 thru 0x030640 (0xfce0 bytes) 000270 1713296815.553750 mapread 0x0135e5 thru 0x018c46 (0x5662 bytes) 000271 1713296815.555308 MIRROR_EXTEND to 7 mirrors 000272 1713296815.647723 mapread 0x0279d1 thru 0x03137a (0x99aa bytes) 000273 1713296815.654562 MIRROR_EXTEND to 8 mirrors 000275 1713296815.751369 write 0x0323e7 thru 0x0381d1 (0x5deb bytes) 000276 1713296815.766489 read 0x025408 thru 0x02b5fc (0x61f5 bytes) 000279 1713296815.769254 trunc from 0x040000 to 0x02b732 (0x148cd bytes) 000280 1713296815.777675 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000281 1713296815.814550 read 0x002e27 thru 0x012c54 (0xfe2e bytes) 000283 1713296815.817115 mapwrite 0x014d49 thru 0x0153fb (0x6b3 bytes) 000284 1713296815.832273 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000285 1713296815.872954 mapwrite 0x000be2 thru 0x00fab8 (0xeed7 bytes) 000286 1713296815.890152 mapwrite 0x022237 thru 0x0289ce (0x6798 bytes) 000287 1713296815.903475 mapread 0x022847 thru 0x02b732 (0x8eec bytes) 000288 1713296815.905038 trunc from 0x02b733 to 0x02f9a6 (0x4274 bytes) 000289 1713296815.909147 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000290 1713296815.944509 mapread 0x0285bd thru 0x029050 (0xa94 bytes) 000291 1713296815.946815 mapread 0x00afbf thru 0x00e5e4 (0x3626 bytes) 000292 1713296815.948634 mapwrite 0x027c48 thru 0x0345a5 (0xc95e bytes) 000294 1713296815.970873 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000296 1713296816.011063 MIRROR_RESYNC 4 mirrors 000297 1713296816.136209 write 0x036922 thru 0x03ffff (0x96de bytes) 000301 1713296816.148256 mapread 0x00f78d thru 0x01f4d6 (0xfd4a bytes) 000302 1713296816.153378 mapwrite 0x035252 thru 0x03e1ac (0x8f5b bytes) 000307 1713296816.168039 mapread 0x03a7ca thru 0x03f334 (0x4b6b bytes) 000310 1713296816.168822 mapwrite 0x039b7f thru 0x03ffff (0x6481 bytes) 000311 1713296816.182438 read 0x0362ec thru 0x03ffff (0x9d14 bytes) 000312 1713296816.183984 MIRROR_RESYNC 4 mirrors 000315 1713296816.291654 mapwrite 0x0005b5 thru 0x00f8eb (0xf337 bytes) 000316 1713296816.334135 read 0x03eec2 thru 0x03ffff (0x113e bytes) 000317 1713296816.335716 write 0x02eb93 thru 0x033a81 (0x4eef bytes) 000319 1713296816.338235 write 0x00343e thru 0x007b1e (0x46e1 bytes) 000324 1713296816.340032 mapread 0x015a60 thru 0x023248 (0xd7e9 bytes) 000326 1713296816.343838 trunc from 0x040000 to 0x031804 (0xe7fb bytes) 000327 1713296816.351731 mapwrite 0x02213a thru 0x026f30 (0x4df7 bytes) 000328 1713296816.365472 write 0x0399c4 thru 0x03c8e9 (0x2f26 bytes) 000330 1713296816.366558 mapread 0x020af9 thru 0x02f907 (0xee0f bytes) 000331 1713296816.367195 read 0x015140 thru 0x01daa7 (0x8968 bytes) 000332 1713296816.367602 write 0x028336 thru 0x033155 (0xae20 bytes) 000333 1713296816.369835 mapread 0x026cc6 thru 0x029774 (0x2aaf bytes) 000335 1713296816.370702 mapread 0x02a53f thru 0x038f38 (0xe9fa bytes) 000336 1713296816.372681 mapread 0x028a11 thru 0x0374bc (0xeaac bytes) 000337 1713296816.373251 mapwrite 0x029e3d thru 0x02ad4b (0xf0f bytes) 000340 1713296816.385499 trunc from 0x03c8ea to 0x011a88 (0x2ae61 bytes) 000341 1713296816.389946 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000342 1713296816.427870 MIRROR_RESYNC 3 mirrors 000343 1713296816.519930 mapread 0x005489 thru 0x00bf9e (0x6b16 bytes) 000350 1713296816.526098 MIRROR_RESYNC 3 mirrors 000352 1713296816.562629 MIRROR_RESYNC 3 mirrors 000353 1713296816.592928 trunc from 0x011a89 to 0x0080b0 (0x99d8 bytes) 000354 1713296816.608421 trunc from 0x0080b1 to 0x038ccf (0x30c1f bytes) 000355 1713296816.612316 trunc from 0x038cd0 to 0x036389 (0x2946 bytes) 000356 1713296816.615518 trunc from 0x03638a to 0x0297a6 (0xcbe3 bytes) 000358 1713296816.619119 trunc from 0x0297a7 to 0x0040c6 (0x256e0 bytes) 000359 1713296816.622893 read 0x00212a thru 0x0040c6 (0x1f9d bytes) 000360 1713296816.624455 mapread 0x002f7e thru 0x004039 (0x10bc bytes) 000362 1713296816.625118 mapwrite 0x0314d8 thru 0x03e366 (0xce8f bytes) 000364 1713296816.650254 MIRROR_RESYNC 3 mirrors 000369 1713296816.748244 MIRROR_EXTEND to 4 mirrors 000371 1713296816.839970 mapread 0x035cd5 thru 0x03e366 (0x8692 bytes) 000374 1713296816.844539 read 0x02bb27 thru 0x02c371 (0x84b bytes) 000377 1713296816.845747 mapwrite 0x02516e thru 0x03375a (0xe5ed bytes) 000379 1713296816.892698 mapwrite 0x030291 thru 0x037f93 (0x7d03 bytes) 000380 1713296816.906734 mapwrite 0x02afa5 thru 0x02b6e9 (0x745 bytes) 000386 1713296816.919631 mapread 0x015835 thru 0x021b03 (0xc2cf bytes) 000387 1713296816.923995 mapread 0x02bb68 thru 0x02f1a5 (0x363e bytes) 000388 1713296816.924494 mapwrite 0x032a6f thru 0x03c08e (0x9620 bytes) 000390 1713296816.938058 read 0x01400a thru 0x01ddad (0x9da4 bytes) 000391 1713296816.939668 MIRROR_EXTEND to 5 mirrors 000392 1713296817.029945 mapwrite 0x01bff1 thru 0x0280b1 (0xc0c1 bytes) 000394 1713296817.076634 mapwrite 0x03c629 thru 0x03ffff (0x39d7 bytes) 000397 1713296817.098736 write 0x0111b6 thru 0x01c781 (0xb5cc bytes) 000399 1713296817.101512 write 0x01f68d thru 0x029e70 (0xa7e4 bytes) 000400 1713296817.102103 write 0x02a043 thru 0x03596a (0xb928 bytes) 000402 1713296817.102966 read 0x02007c thru 0x02308e (0x3013 bytes) 000403 1713296817.103338 trunc from 0x040000 to 0x005d87 (0x3a278 bytes) 000404 1713296817.106941 trunc from 0x005d88 to 0x010068 (0xa2e1 bytes) 000405 1713296817.109644 write 0x00acfe thru 0x0126d3 (0x79d6 bytes) 000407 1713296817.112219 mapwrite 0x02d2da thru 0x037b83 (0xa8aa bytes) 000408 1713296817.132838 read 0x013ec7 thru 0x022d37 (0xee71 bytes) 000409 1713296817.135912 mapwrite 0x0353ff thru 0x038f85 (0x3b87 bytes) 000410 1713296817.153423 mapread 0x000157 thru 0x00bb13 (0xb9bd bytes) 000413 1713296817.154780 trunc from 0x038f86 to 0x01cdae (0x1c1d7 bytes) 000414 1713296817.158551 trunc from 0x01cdaf to 0x02436b (0x75bd bytes) 000415 1713296817.161713 write 0x010a02 thru 0x0149f6 (0x3ff5 bytes) 000416 1713296817.162227 mapwrite 0x0140a7 thru 0x01f25f (0xb1b9 bytes) 000417 1713296817.179081 read 0x0081c7 thru 0x015606 (0xd440 bytes) 000418 1713296817.180466 MIRROR_EXTEND to 6 mirrors 000419 1713296817.276152 MIRROR_RESYNC 6 mirrors 000420 1713296817.407102 mapread 0x022d11 thru 0x02436b (0x165b bytes) 000421 1713296817.409747 mapread 0x020943 thru 0x02436b (0x3a29 bytes) 000422 1713296817.411897 mapread 0x01269b thru 0x018282 (0x5be8 bytes) 000424 1713296817.414062 write 0x026ace thru 0x02f161 (0x8694 bytes) 000425 1713296817.428681 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000426 1713296817.471069 MIRROR_RESYNC 5 mirrors 000427 1713296817.614545 mapwrite 0x00ab67 thru 0x012a9c (0x7f36 bytes) 000431 1713296817.660749 mapwrite 0x03bc27 thru 0x03ffff (0x43d9 bytes) 000432 1713296817.683181 mapread 0x02ba34 thru 0x02fc12 (0x41df bytes) 000434 1713296817.685587 read 0x0040dd thru 0x005dbe (0x1ce2 bytes) 000435 1713296817.686734 MIRROR_EXTEND to 6 mirrors 000438 1713296817.788682 mapread 0x00a53c thru 0x00bacb (0x1590 bytes) 000440 1713296817.793752 MIRROR_EXTEND to 7 mirrors 000443 1713296817.898553 write 0x011f25 thru 0x0143f8 (0x24d4 bytes) 000444 1713296817.911039 mapwrite 0x03762c thru 0x03c7c3 (0x5198 bytes) 000446 1713296817.926525 MIRROR_EXTEND to 8 mirrors 000448 1713296818.022917 mapread 0x001013 thru 0x003881 (0x286f bytes) 000451 1713296818.029519 mapwrite 0x03eec4 thru 0x03ffff (0x113c bytes) 000452 1713296818.070175 mapread 0x00cbae thru 0x00e73d (0x1b90 bytes) 000453 1713296818.072519 mapwrite 0x032688 thru 0x03e234 (0xbbad bytes) 000454 1713296818.086617 trunc from 0x040000 to 0x01ea7e (0x21581 bytes) 000459 1713296818.096253 mapread 0x00b5af thru 0x00db6a (0x25bc bytes) 000460 1713296818.096887 read 0x003f84 thru 0x011bb4 (0xdc31 bytes) 000464 1713296818.099399 mapwrite 0x016e94 thru 0x022bc8 (0xbd35 bytes) 000467 1713296818.119033 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000468 1713296818.158470 mapwrite 0x0160dc thru 0x01e12d (0x8052 bytes) 000469 1713296818.172733 trunc from 0x022bc9 to 0x02b37a (0x87b2 bytes) 000470 1713296818.175976 mapread 0x0051bb thru 0x009577 (0x43bd bytes) 000472 1713296818.178288 mapwrite 0x02b1b2 thru 0x0316ff (0x654e bytes) truncating to largest ever: 0x3cb45 000473 1713296818.193766 trunc from 0x031700 to 0x03cb44 (0xb445 bytes) 000476 1713296818.197053 mapwrite 0x0253ac thru 0x02fb3b (0xa790 bytes) 000479 1713296818.209419 trunc from 0x03cb45 to 0x002683 (0x3a4c1 bytes) 000480 1713296818.213771 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000481 1713296818.250992 write 0x034a77 thru 0x03ffff (0xb589 bytes) 000482 1713296818.252099 trunc from 0x040000 to 0x013c35 (0x2c3ca bytes) 000483 1713296818.256505 MIRROR_EXTEND to 7 mirrors 000484 1713296818.343612 MIRROR_EXTEND to 8 mirrors 000485 1713296818.435969 MIRROR_RESYNC 8 mirrors 000487 1713296818.596066 write 0x011c9f thru 0x01e904 (0xcc66 bytes) 000488 1713296818.610047 trunc from 0x01e905 to 0x026c12 (0x830e bytes) 000492 1713296818.619381 MIRROR_EXTEND to 9 mirrors 000495 1713296818.725809 read 0x01cc39 thru 0x02045b (0x3823 bytes) 000496 1713296818.728291 read 0x00c63a thru 0x00e40f (0x1dd6 bytes) 000498 1713296818.731904 mapwrite 0x03b7de thru 0x03ffff (0x4822 bytes) 000499 1713296818.765666 trunc from 0x040000 to 0x037ff3 (0x800c bytes) 000500 1713296818.770267 write 0x01fc0e thru 0x029d45 (0xa138 bytes) 000501 1713296818.772200 mapread 0x0064dc thru 0x00a994 (0x44b9 bytes) 000502 1713296818.777123 write 0x00e72f thru 0x00fe90 (0x1762 bytes) 000504 1713296818.779565 mapread 0x010dd8 thru 0x020aed (0xfd16 bytes) 000507 1713296818.792754 mapread 0x02c97f thru 0x02f685 (0x2d07 bytes) 000508 1713296818.793352 mapread 0x00a753 thru 0x010084 (0x5932 bytes) 000511 1713296818.796755 trunc from 0x037ff4 to 0x011471 (0x26b82 bytes) 000513 1713296818.800418 mapread 0x007fee thru 0x00d874 (0x5887 bytes) 000514 1713296818.800879 trunc from 0x011472 to 0x0201df (0xed6e bytes) 000515 1713296818.804000 write 0x022d41 thru 0x02c347 (0x9607 bytes) 000517 1713296818.806292 MIRROR_EXTEND to 10 mirrors 000518 1713296818.896489 mapread 0x026c1d thru 0x02a641 (0x3a25 bytes) 000520 1713296818.902026 read 0x009c55 thru 0x0153a5 (0xb751 bytes) 000524 1713296818.906959 read 0x006b96 thru 0x0136a1 (0xcb0c bytes) 000525 1713296818.909430 read 0x00037f thru 0x00d890 (0xd512 bytes) 000526 1713296818.910357 read 0x01eadb thru 0x028aa2 (0x9fc8 bytes) 000527 1713296818.911177 mapread 0x027be7 thru 0x02c347 (0x4761 bytes) 000529 1713296818.913712 write 0x011b5f thru 0x01353d (0x19df bytes) 000530 1713296818.928552 read 0x02aee3 thru 0x02c347 (0x1465 bytes) 000533 1713296818.929540 read 0x023248 thru 0x0284d8 (0x5291 bytes) 000535 1713296818.930043 mapread 0x012986 thru 0x01f258 (0xc8d3 bytes) 000540 1713296818.932497 MIRROR_RESYNC 10 mirrors 000541 1713296819.183618 MIRROR_RESYNC 10 mirrors 000542 1713296819.232192 read 0x0077c5 thru 0x0147da (0xd016 bytes) 000543 1713296819.234356 write 0x033c47 thru 0x03ffff (0xc3b9 bytes) 000547 1713296819.247474 read 0x03e19a thru 0x03ffff (0x1e66 bytes) 000549 1713296819.247981 mapread 0x0049e7 thru 0x009f2b (0x5545 bytes) 000551 1713296819.250738 read 0x01e026 thru 0x02257d (0x4558 bytes) 000552 1713296819.252065 MIRROR_EXTEND to 11 mirrors 000554 1713296819.364530 read 0x01edb3 thru 0x021e76 (0x30c4 bytes) 000555 1713296819.366650 mapwrite 0x01064d thru 0x01bb30 (0xb4e4 bytes) 000557 1713296819.409792 read 0x02adc6 thru 0x033b9b (0x8dd6 bytes) 000558 1713296819.412444 mapwrite 0x03ef85 thru 0x03ffff (0x107b bytes) 000559 1713296819.423205 read 0x022269 thru 0x031b62 (0xf8fa bytes) 000560 1713296819.423565 trunc from 0x040000 to 0x0076d3 (0x3892c bytes) 000562 1713296819.431507 mapread 0x006147 thru 0x0072f6 (0x11b0 bytes) 000563 1713296819.432020 read 0x004bcc thru 0x005d46 (0x117b bytes) 000564 1713296819.432413 mapread 0x00445e thru 0x0076d3 (0x3276 bytes) 000565 1713296819.432945 trunc from 0x0076d4 to 0x03945e (0x31d8b bytes) 000567 1713296819.436873 write 0x00f02d thru 0x016198 (0x716c bytes) 000569 1713296819.440793 mapread 0x01710f thru 0x024dce (0xdcc0 bytes) 000572 1713296819.441963 write 0x00aff5 thru 0x00e192 (0x319e bytes) 000573 1713296819.442504 mapwrite 0x001ca7 thru 0x006dd0 (0x512a bytes) 000575 1713296819.459595 mapwrite 0x017c1a thru 0x019803 (0x1bea bytes) 000576 1713296819.471767 mapwrite 0x01226b thru 0x016726 (0x44bc bytes) 000578 1713296819.484503 read 0x00ac92 thru 0x01089a (0x5c09 bytes) 000582 1713296819.485826 write 0x0060a5 thru 0x0071ba (0x1116 bytes) 000585 1713296819.487946 MIRROR_RESYNC 11 mirrors 000586 1713296819.737535 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000587 1713296819.805535 trunc from 0x03945f to 0x016843 (0x22c1b bytes) 000588 1713296819.822862 read 0x00c602 thru 0x016843 (0xa242 bytes) 000589 1713296819.824768 trunc from 0x016844 to 0x03a043 (0x23800 bytes) 000590 1713296819.828553 mapread 0x00115b thru 0x00b275 (0xa11b bytes) 000594 1713296819.833978 mapwrite 0x03a6e4 thru 0x03c9cd (0x22ea bytes) 000597 1713296819.853272 trunc from 0x03c9ce to 0x02f194 (0xd839 bytes) 000600 1713296819.857283 read 0x004c3b thru 0x00fe4b (0xb211 bytes) 000601 1713296819.858839 MIRROR_RESYNC 10 mirrors 000604 1713296820.099296 write 0x0212bf thru 0x022aa1 (0x17e3 bytes) 000605 1713296820.113530 write 0x035583 thru 0x03d3a8 (0x7e26 bytes) 000607 1713296820.115841 MIRROR_EXTEND to 11 mirrors 000608 1713296820.221784 MIRROR_EXTEND to 12 mirrors 000609 1713296820.323082 write 0x0379c1 thru 0x03ffff (0x863f bytes) 000611 1713296820.338252 MIRROR_RESYNC 12 mirrors 000614 1713296820.574013 write 0x01ed8a thru 0x02aeea (0xc161 bytes) 000617 1713296820.588114 mapread 0x02cafb thru 0x03c94b (0xfe51 bytes) 000618 1713296820.592877 mapread 0x022e31 thru 0x026b32 (0x3d02 bytes) 000623 1713296820.594244 trunc from 0x040000 to 0x02226d (0x1dd92 bytes) 000624 1713296820.598756 mapwrite 0x03a53f thru 0x03ffff (0x5ac1 bytes) 000626 1713296820.617396 mapwrite 0x02d5f2 thru 0x0359c6 (0x83d5 bytes) 000627 1713296820.634265 mapread 0x00adab thru 0x0107ff (0x5a55 bytes) 000630 1713296820.637781 MIRROR_EXTEND to 13 mirrors 000631 1713296820.743313 mapread 0x02c082 thru 0x038c6a (0xcbe9 bytes) 000633 1713296820.752318 write 0x0092a5 thru 0x009ea3 (0xbff bytes) 000635 1713296820.766630 mapread 0x0059b1 thru 0x01174d (0xbd9d bytes) 000636 1713296820.767297 mapread 0x006a48 thru 0x009486 (0x2a3f bytes) 000637 1713296820.767625 read 0x00c21b thru 0x013779 (0x755f bytes) 000638 1713296820.769014 write 0x02f82d thru 0x02fe21 (0x5f5 bytes) 000639 1713296820.771146 read 0x01d555 thru 0x01ed54 (0x1800 bytes) 000640 1713296820.773409 read 0x0135f3 thru 0x013752 (0x160 bytes) 000642 1713296820.775007 MIRROR_RESYNC 13 mirrors 000643 1713296821.024828 write 0x00ad98 thru 0x0131a3 (0x840c bytes) 000645 1713296821.038721 mapread 0x009370 thru 0x011c86 (0x8917 bytes) 000646 1713296821.040990 trunc from 0x040000 to 0x0049dd (0x3b622 bytes) 000647 1713296821.048143 trunc from 0x0049de to 0x02d2fb (0x2891e bytes) 000648 1713296821.052307 MIRROR_EXTEND to 14 mirrors 000650 1713296821.155351 read 0x021a71 thru 0x0288ff (0x6e8f bytes) 000651 1713296821.159008 MIRROR_RESYNC 14 mirrors 000652 1713296821.446840 read 0x01db3d thru 0x025458 (0x791c bytes) 000653 1713296821.449493 MIRROR_RESYNC 14 mirrors 000658 1713296821.508146 MIRROR_RESYNC 14 mirrors 000662 1713296821.564424 read 0x022f0a thru 0x02ad55 (0x7e4c bytes) 000663 1713296821.567085 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000664 1713296821.642087 write 0x0322ef thru 0x03eac5 (0xc7d7 bytes) 000669 1713296821.657097 read 0x033be6 thru 0x03d593 (0x99ae bytes) 000670 1713296821.657567 mapwrite 0x018987 thru 0x028856 (0xfed0 bytes) 000671 1713296821.680788 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000674 1713296821.721930 mapwrite 0x00148e thru 0x0035a8 (0x211b bytes) 000675 1713296821.738279 mapread 0x0085ab thru 0x00888c (0x2e2 bytes) 000676 1713296821.738915 write 0x024dbb thru 0x024e98 (0xde bytes) 000677 1713296821.741580 write 0x0367ec thru 0x03ffff (0x9814 bytes) 000680 1713296821.745513 MIRROR_RESYNC 12 mirrors 000681 1713296821.978565 trunc from 0x040000 to 0x0223a7 (0x1dc58 bytes) 000682 1713296821.997774 write 0x00529a thru 0x00a8b5 (0x561c bytes) 000684 1713296822.001457 read 0x004abf thru 0x00d4c6 (0x8a08 bytes) 000685 1713296822.003524 write 0x03db8f thru 0x03ffff (0x2471 bytes) 000688 1713296822.004565 trunc from 0x040000 to 0x008031 (0x37fce bytes) 000689 1713296822.008514 mapread 0x001b82 thru 0x008031 (0x64b0 bytes) 000690 1713296822.010184 write 0x025cb6 thru 0x02b3db (0x5726 bytes) 000692 1713296822.011775 mapread 0x015b67 thru 0x016225 (0x6bf bytes) 000696 1713296822.014824 trunc from 0x02b3dc to 0x039845 (0xe46a bytes) 000697 1713296822.018608 mapread 0x0092ec thru 0x012547 (0x925c bytes) 000698 1713296822.021006 read 0x031ade thru 0x033ded (0x2310 bytes) 000703 1713296822.024708 mapread 0x0120c7 thru 0x01f16e (0xd0a8 bytes) 000704 1713296822.025330 trunc from 0x039846 to 0x009efc (0x2f949 bytes) 000705 1713296822.029297 read 0x0084e0 thru 0x009efc (0x1a1d bytes) 000706 1713296822.029619 trunc from 0x009efd to 0x00917c (0xd80 bytes) 000708 1713296822.032948 read 0x0023c0 thru 0x005fe7 (0x3c28 bytes) 000709 1713296822.034257 MIRROR_RESYNC 12 mirrors 000711 1713296822.293648 MIRROR_RESYNC 12 mirrors 000714 1713296822.339267 read 0x003e75 thru 0x007044 (0x31d0 bytes) 000715 1713296822.341210 write 0x036761 thru 0x03f077 (0x8917 bytes) 000716 1713296822.356217 MIRROR_EXTEND to 13 mirrors 000717 1713296822.456529 mapwrite 0x008f89 thru 0x0137ca (0xa842 bytes) 000719 1713296822.500537 read 0x020810 thru 0x021a62 (0x1253 bytes) 000721 1713296822.502539 write 0x02bf6c thru 0x02cfcc (0x1061 bytes) 000723 1713296822.505134 write 0x00d91b thru 0x0124d1 (0x4bb7 bytes) 000725 1713296822.506048 mapread 0x008c5c thru 0x00ab5c (0x1f01 bytes) 000726 1713296822.506454 mapwrite 0x0340d2 thru 0x03b169 (0x7098 bytes) 000730 1713296822.521074 write 0x00eec6 thru 0x019631 (0xa76c bytes) 000731 1713296822.521738 trunc from 0x03f078 to 0x002b78 (0x3c4ff bytes) 000732 1713296822.530330 write 0x000ba6 thru 0x003edd (0x3338 bytes) 000737 1713296822.533992 MIRROR_EXTEND to 14 mirrors 000740 1713296822.621062 write 0x01b821 thru 0x025daa (0xa58a bytes) 000741 1713296822.634693 write 0x00de14 thru 0x014115 (0x6302 bytes) 000743 1713296822.637609 mapwrite 0x02292e thru 0x02d825 (0xaef8 bytes) 000745 1713296822.661807 trunc from 0x02d826 to 0x01d745 (0x100e0 bytes) 000746 1713296822.665225 trunc from 0x01d746 to 0x0172e7 (0x645e bytes) 000747 1713296822.668131 read 0x016652 thru 0x01695f (0x30e bytes) 000749 1713296822.669490 trunc from 0x0172e8 to 0x02b6d8 (0x143f1 bytes) truncating to largest ever: 0x3e2fd 000750 1713296822.672264 trunc from 0x02b6d9 to 0x03e2fc (0x12c24 bytes) 000753 1713296822.675280 mapwrite 0x0105e4 thru 0x012a06 (0x2423 bytes) 000754 1713296822.687619 mapwrite 0x018d46 thru 0x01909c (0x357 bytes) 000756 1713296822.702343 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000757 1713296822.742291 write 0x005ebb thru 0x01568d (0xf7d3 bytes) 000759 1713296822.745132 trunc from 0x03e2fd to 0x013f01 (0x2a3fb bytes) 000763 1713296822.749297 read 0x0024f5 thru 0x002e12 (0x91e bytes) 000764 1713296822.750540 write 0x000171 thru 0x005e33 (0x5cc3 bytes) 000765 1713296822.752404 trunc from 0x013f02 to 0x030b64 (0x1cc63 bytes) 000766 1713296822.756243 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000770 1713296822.800921 read 0x02c6b6 thru 0x030b64 (0x44af bytes) 000771 1713296822.803048 write 0x024643 thru 0x027e6b (0x3829 bytes) 000775 1713296822.805585 read 0x02fe9f thru 0x030b64 (0xcc6 bytes) 000778 1713296822.807092 MIRROR_RESYNC 12 mirrors 000779 1713296823.096449 mapwrite 0x014e91 thru 0x0175cd (0x273d bytes) 000782 1713296823.139639 trunc from 0x030b65 to 0x020207 (0x1095d bytes) 000783 1713296823.143721 mapwrite 0x0294f1 thru 0x02b5e8 (0x20f8 bytes) 000786 1713296823.160843 trunc from 0x02b5e9 to 0x007a0b (0x23bdd bytes) 000787 1713296823.163871 trunc from 0x007a0c to 0x037f7b (0x30570 bytes) 000790 1713296823.167360 trunc from 0x037f7c to 0x02b0c4 (0xceb7 bytes) 000791 1713296823.170950 MIRROR_EXTEND to 13 mirrors 000792 1713296823.267019 write 0x00ba33 thru 0x015eb3 (0xa481 bytes) 000793 1713296823.281687 read 0x01ee0b thru 0x023dce (0x4fc4 bytes) 000795 1713296823.284953 MIRROR_EXTEND to 14 mirrors 000796 1713296823.386577 read 0x00cc3d thru 0x0183c9 (0xb78d bytes) 000797 1713296823.389592 write 0x006333 thru 0x0134fa (0xd1c8 bytes) 000798 1713296823.405951 MIRROR_RESYNC 14 mirrors 000799 1713296823.714544 read 0x00f3de thru 0x018efb (0x9b1e bytes) 000800 1713296823.716433 write 0x006e87 thru 0x010603 (0x977d bytes) 000801 1713296823.729530 mapread 0x01f1de thru 0x02a1a4 (0xafc7 bytes) 000802 1713296823.733013 read 0x01ea1b thru 0x01ee13 (0x3f9 bytes) 000803 1713296823.733274 trunc from 0x02b0c5 to 0x00b8f3 (0x1f7d1 bytes) 000806 1713296823.740282 mapread 0x00410f thru 0x0059ab (0x189d bytes) 000807 1713296823.742136 mapwrite 0x004e31 thru 0x00d172 (0x8342 bytes) 000808 1713296823.762069 trunc from 0x00d173 to 0x023e4b (0x16cd9 bytes) 000809 1713296823.765249 read 0x00835d thru 0x00b642 (0x32e6 bytes) 000810 1713296823.765650 read 0x008950 thru 0x008d78 (0x429 bytes) 000811 1713296823.766021 read 0x0038e3 thru 0x012d94 (0xf4b2 bytes) 000812 1713296823.769781 MIRROR_EXTEND to 15 mirrors 000814 1713296823.867884 MIRROR_RESYNC 15 mirrors 000816 1713296824.202490 write 0x035d5a thru 0x03d0bb (0x7362 bytes) 000817 1713296824.214667 read 0x01aa05 thru 0x0233d2 (0x89ce bytes) 000818 1713296824.216958 trunc from 0x03d0bc to 0x01b962 (0x21759 bytes) 000819 1713296824.224314 write 0x03cf86 thru 0x03ffff (0x307a bytes) 000820 1713296824.224902 mapwrite 0x032163 thru 0x03ffff (0xde9d bytes) 000822 1713296824.245133 mapwrite 0x02faf3 thru 0x032f67 (0x3475 bytes) 000823 1713296824.257615 read 0x038d4b thru 0x03f445 (0x66fb bytes) 000824 1713296824.258082 mapread 0x0269a6 thru 0x0293c3 (0x2a1e bytes) 000826 1713296824.261220 write 0x01cef2 thru 0x02556c (0x867b bytes) 000831 1713296824.262389 read 0x03a569 thru 0x03ffff (0x5a97 bytes) 000832 1713296824.262614 mapwrite 0x0111b3 thru 0x0203d4 (0xf222 bytes) 000833 1713296824.276514 mapwrite 0x03e641 thru 0x03f383 (0xd43 bytes) 000834 1713296824.288263 mapwrite 0x03f077 thru 0x03ffff (0xf89 bytes) 000835 1713296824.298557 read 0x002028 thru 0x005e30 (0x3e09 bytes) 000837 1713296824.299297 read 0x03edaf thru 0x03ffff (0x1251 bytes) 000839 1713296824.301514 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000841 1713296824.347597 read 0x00dfdc thru 0x01bd2e (0xdd53 bytes) 000842 1713296824.349685 mapread 0x01165b thru 0x01ba59 (0xa3ff bytes) 000843 1713296824.350104 mapread 0x0366b2 thru 0x03ffff (0x994e bytes) 000845 1713296824.353725 mapwrite 0x039b39 thru 0x03ffff (0x64c7 bytes) 000846 1713296824.364933 trunc from 0x040000 to 0x02f199 (0x10e66 bytes) 000847 1713296824.368593 write 0x012670 thru 0x013c15 (0x15a6 bytes) 000848 1713296824.369062 mapread 0x0110b6 thru 0x0134cb (0x2416 bytes) 000850 1713296824.369622 read 0x02767d thru 0x02d3f0 (0x5d74 bytes) 000851 1713296824.372048 write 0x01903f thru 0x01ff87 (0x6f49 bytes) truncating to largest ever: 0x3fdbf 000852 1713296824.372519 trunc from 0x02f19a to 0x03fdbe (0x10c25 bytes) 000853 1713296824.375835 read 0x02bda1 thru 0x03aa46 (0xeca6 bytes) 000854 1713296824.378278 read 0x012790 thru 0x01a2a0 (0x7b11 bytes) 000855 1713296824.378636 mapread 0x003aa0 thru 0x00964b (0x5bac bytes) 000856 1713296824.379234 trunc from 0x03fdbf to 0x0341b0 (0xbc0e bytes) 000857 1713296824.382382 mapread 0x01652d thru 0x01c30d (0x5de1 bytes) 000859 1713296824.383061 write 0x022eac thru 0x02596c (0x2ac1 bytes) 000860 1713296824.383498 trunc from 0x0341b1 to 0x01d3cf (0x16de1 bytes) 000863 1713296824.387518 MIRROR_RESYNC 14 mirrors 000865 1713296824.667689 read 0x010a26 thru 0x018f9d (0x8578 bytes) 000866 1713296824.669363 read 0x00919a thru 0x012deb (0x9c52 bytes) 000868 1713296824.671274 read 0x00b91d thru 0x012d08 (0x73ec bytes) 000869 1713296824.671517 read 0x009060 thru 0x012002 (0x8fa3 bytes) 000872 1713296824.672906 MIRROR_EXTEND to 15 mirrors 000875 1713296824.780622 read 0x012aae thru 0x017ac7 (0x501a bytes) 000876 1713296824.783070 MIRROR_EXTEND to 16 mirrors 000877 1713296824.897906 read 0x0029b1 thru 0x00bc5b (0x92ab bytes) 000878 1713296824.899587 mapwrite 0x020d59 thru 0x024e4a (0x40f2 bytes) 000879 1713296824.933118 write 0x0068e4 thru 0x00fae8 (0x9205 bytes) 000880 1713296824.935487 mapread 0x00a1df thru 0x0110c3 (0x6ee5 bytes) 000881 1713296824.937428 trunc from 0x024e4b to 0x0124fa (0x12950 bytes) 000885 1713296824.941706 trunc from 0x0124fb to 0x009853 (0x8ca7 bytes) 000890 1713296824.946113 mapwrite 0x03b42c thru 0x03c5b5 (0x118a bytes) 000893 1713296824.963791 mapread 0x00f576 thru 0x018483 (0x8f0e bytes) 000895 1713296824.967495 mapread 0x02d2f2 thru 0x035d65 (0x8a74 bytes) 000896 1713296824.969451 write 0x0106a2 thru 0x01c4a5 (0xbe04 bytes) 000899 1713296824.970552 read 0x00cae8 thru 0x00f51d (0x2a36 bytes) 000900 1713296824.970826 write 0x03e9e8 thru 0x03ffff (0x1618 bytes) 000901 1713296824.971196 trunc from 0x040000 to 0x011e4b (0x2e1b4 bytes) 000902 1713296824.974581 mapread 0x00aee2 thru 0x00c0a0 (0x11bf bytes) 000903 1713296824.975029 trunc from 0x011e4c to 0x0267cb (0x14980 bytes) 000904 1713296824.977822 write 0x0297fe thru 0x03439b (0xab9e bytes) 000905 1713296824.979428 MIRROR_RESYNC 16 mirrors 000906 1713296825.328707 mapread 0x03194e thru 0x03227b (0x92e bytes) 000907 1713296825.330777 mapread 0x01fd19 thru 0x02363e (0x3926 bytes) 000908 1713296825.332493 read 0x026966 thru 0x033a72 (0xd10d bytes) 000909 1713296825.334813 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000911 1713296825.417209 write 0x036d40 thru 0x03ca58 (0x5d19 bytes) 000913 1713296825.431147 mapwrite 0x03521d thru 0x03cc67 (0x7a4b bytes) 000914 1713296825.459936 write 0x01625f thru 0x01765f (0x1401 bytes) 000915 1713296825.462812 write 0x0361e3 thru 0x03b982 (0x57a0 bytes) 000916 1713296825.463246 mapwrite 0x002bae thru 0x007455 (0x48a8 bytes) 000918 1713296825.478609 read 0x02504d thru 0x028f8a (0x3f3e bytes) 000919 1713296825.480276 mapread 0x02ae47 thru 0x03a355 (0xf50f bytes) 000920 1713296825.483425 trunc from 0x03cc68 to 0x0193e9 (0x2387e bytes) 000921 1713296825.488011 MIRROR_EXTEND to 16 mirrors 000924 1713296825.578237 write 0x01d57e thru 0x01fc8b (0x270e bytes) 000925 1713296825.591996 mapwrite 0x01405a thru 0x016233 (0x21da bytes) 000926 1713296825.611669 trunc from 0x01fc8c to 0x008681 (0x1760a bytes) 000927 1713296825.618809 trunc from 0x008682 to 0x00a8cb (0x224a bytes) 000928 1713296825.621619 mapwrite 0x01803d thru 0x022e6a (0xae2e bytes) 000929 1713296825.642005 trunc from 0x022e6b to 0x00addc (0x1808e bytes) 000931 1713296825.646562 MIRROR_RESYNC 16 mirrors 000935 1713296825.957166 mapread 0x00a576 thru 0x00addc (0x867 bytes) 000936 1713296825.957537 mapwrite 0x021174 thru 0x02c851 (0xb6de bytes) 000937 1713296825.993724 write 0x02ad9a thru 0x039eb0 (0xf117 bytes) 000938 1713296825.994521 mapread 0x036225 thru 0x039eb0 (0x3c8c bytes) 000940 1713296825.995089 mapread 0x023277 thru 0x028583 (0x530d bytes) 000941 1713296825.995441 write 0x015e0b thru 0x02328a (0xd480 bytes) 000943 1713296825.998073 write 0x003970 thru 0x003e43 (0x4d4 bytes) 000944 1713296825.999793 mapwrite 0x011da3 thru 0x020986 (0xebe4 bytes) 000945 1713296826.016712 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000947 1713296826.056543 read 0x014fd8 thru 0x0240de (0xf107 bytes) 000948 1713296826.058678 read 0x0269b8 thru 0x028e5c (0x24a5 bytes) 000950 1713296826.062545 MIRROR_RESYNC 15 mirrors 000951 1713296826.385721 write 0x030a17 thru 0x034237 (0x3821 bytes) 000953 1713296826.399157 mapwrite 0x015ab9 thru 0x01d349 (0x7891 bytes) 000954 1713296826.415152 read 0x02ebf2 thru 0x039eb0 (0xb2bf bytes) 000955 1713296826.417504 write 0x0207f7 thru 0x0231cf (0x29d9 bytes) 000956 1713296826.419222 read 0x00d81d thru 0x00df58 (0x73c bytes) 000957 1713296826.421273 write 0x000a49 thru 0x000efb (0x4b3 bytes) 000958 1713296826.421649 read 0x031975 thru 0x036b1e (0x51aa bytes) 000960 1713296826.422189 trunc from 0x039eb1 to 0x006130 (0x33d80 bytes) 000962 1713296826.430168 mapread 0x002708 thru 0x004c94 (0x258d bytes) 000963 1713296826.430636 read 0x000fbe thru 0x006130 (0x5173 bytes) 000964 1713296826.431034 mapread 0x003741 thru 0x0056d4 (0x1f94 bytes) 000965 1713296826.432714 MIRROR_RESYNC 15 mirrors 000966 1713296826.718473 write 0x0395a7 thru 0x03e4b8 (0x4f12 bytes) 000968 1713296826.730892 trunc from 0x03e4b9 to 0x02e2ed (0x101cb bytes) 000969 1713296826.738028 mapwrite 0x01d50b thru 0x0258c6 (0x83bc bytes) 000970 1713296826.757774 write 0x02c1b1 thru 0x02f79c (0x35ec bytes) 000971 1713296826.761029 MIRROR_RESYNC 15 mirrors 000973 1713296827.087962 MIRROR_RESYNC 15 mirrors 000974 1713296827.149397 mapread 0x009cf3 thru 0x010a02 (0x6d10 bytes) 000979 1713296827.153775 mapwrite 0x038cb3 thru 0x03aa04 (0x1d52 bytes) 000980 1713296827.184411 MIRROR_EXTEND to 16 mirrors 000981 1713296827.277229 write 0x03b075 thru 0x03ffff (0x4f8b bytes) 000982 1713296827.291771 read 0x00a372 thru 0x0195b0 (0xf23f bytes) 000983 1713296827.293970 mapread 0x019876 thru 0x0281a6 (0xe931 bytes) 000986 1713296827.299178 write 0x01bb85 thru 0x024c30 (0x90ac bytes) 000989 1713296827.301253 MIRROR_SPLIT mirror 4 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000992 1713296827.344894 trunc from 0x040000 to 0x035693 (0xa96c bytes) 000996 1713296827.352145 write 0x0276c2 thru 0x02f7da (0x8119 bytes) 000997 1713296827.354606 read 0x00e0bd thru 0x0172fb (0x923f bytes) 000999 1713296827.356727 mapwrite 0x0362a0 thru 0x03b932 (0x5693 bytes) 001000 1713296827.376756 MIRROR_RESYNC 15 mirrors All operations completed A-OK! PASS 70a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 15:47:10 (1713296830) Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre2 Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7753504 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6516896 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7782784 .. Extending file size to 8497760 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8041024 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8541344 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8756832 .. Extending file size to 8700384 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8825120 .. failed Extending file size to 8853664 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9143168 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9015040 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9124160 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 8259 8260 8261 8263 8266 192.168.203.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg359-client.virtnet /mnt/lustre2 (opts:) 192.168.203.159@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg359-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:48:15 (1713296895) Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre2 Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7757536 .. Extending file size to 6755072 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8434848 .. Extending file size to 7765728 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8823296 .. Extending file size to 8775360 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8973952 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8912064 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8934528 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8984800 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9364960 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9307520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 11823 11824 11825 11827 11830 192.168.203.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg359-client.virtnet /mnt/lustre2 (opts:) 192.168.203.159@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg359-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:49:20 (1713296960) Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5376 2203136 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5376 2203136 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0316806 s, 66.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.080569 s, 26.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg359-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:49:26 (1713296966) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0250298 s, 83.8 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 15:49:29 (1713296969) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe99:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe99:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xe9c:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:49:32 (1713296972) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.028227 s, 74.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:49:35 (1713296975) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0250453 s, 83.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea0:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:49:39 (1713296979) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0240561 s, 87.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea2:0x0] } PASS 204c (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:50:03 (1713297003) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg359-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161422 s, 65.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea0:0x0] } PASS 204d (42s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:50:47 (1713297047) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg359-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0261697 s, 80.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (38s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:51:27 (1713297087) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg359-server mds-ost sync done. Creating new pool oleg359-server: Pool lustre.test_204f created Adding targets to pool oleg359-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg359-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0237964 s, 88.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg359-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg359-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg359-server: Pool lustre.test_204f destroyed PASS 204f (51s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 15:52:19 (1713297139) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:52:23 (1713297143) oleg359-server: Pool lustre.test_206 created Adding targets to pool oleg359-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg359-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg359-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg359-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg359-server: Pool lustre.test_206 destroyed PASS 206 (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:52:36 (1713297156) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015247 s, 68.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104429 s, 100 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xbe66b115 Mirror 3: 0xbe66b115 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:52:40 (1713297160) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:52:43 (1713297163) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:52:46 (1713297166) Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0150533 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2270000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2270000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2270000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x960000 Read this (4 bytes): BBBB 192.168.203.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg359-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:53:01 (1713297181) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00253588 s, 3.2 MB/s the very first read read with warm cache PASS 209b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 15:53:04 (1713297184) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00829149 s, 126 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x7b1 lmm_fid: [0x200000404:0x7b1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x7b1 lmm_fid: [0x200000404:0x7b1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xead:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835889 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-16 15:53:04.000000000 -0400 Modify: 2024-04-16 15:53:04.000000000 -0400 Change: 2024-04-16 15:53:05.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:53:07 (1713297187) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 759 sec ============ 15:53:09 (1713297189) === sanity-flr: start cleanup 15:53:09 (1713297189) === === sanity-flr: finish cleanup 15:53:09 (1713297189) ===