== sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 11:15:19 (1713280519) ##### The namespace LFSCK will move the orphans from backend /lost+found directory to normal client visible namespace or to global visible ./lustre/lost+found/MDTxxxx/ directory ##### Inject failure stub on MDT0 to simulate the case that directory d0 has no linkEA entry, then the LFSCK will move it into .lustre/lost+found/MDTxxxx/ later. fail_loc=0x161d fail_loc=0 Inject failure stub on MDT0 to simulate the case that the object's name entry will be removed, but not destroy the object. Then backend e2fsck will handle it as orphan and add them into the backend /lost+found directory. fail_loc=0x1624 fail_loc=0 192.168.201.122@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg122-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-mds1 (opts:) on oleg122-server run e2fsck on mds1 e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8 oleg122-server: e2fsck 1.46.2.wc5 (26-Mar-2022) oleg122-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 162 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 163 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 164 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 165 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 167 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 168 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 169 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 170 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 171 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 172 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 173 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 174 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 175 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 176 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 177 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 179 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 182 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 185 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 186 badness 0 to 2 for 10084 [Thread 0] group 1 finished [Thread 0] e2fsck_pass1_run:2564: increase inode 20036 badness 0 to 2 for 10084 [Thread 0] group 2 finished [Thread 0] Pass 1: Memory used: 264k/0k (132k/133k), time: 0.01/ 0.00/ 0.01 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 110.93MB/s [Thread 0] Scanned group range [0, 2), inodes 389 Pass 2: Checking directory structure Pass 2: Memory used: 264k/0k (89k/176k), time: 0.01/ 0.00/ 0.01 Pass 2: I/O read: 1MB, write: 0MB, rate: 143.74MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 264k/0k (89k/176k), time: 0.03/ 0.01/ 0.02 Unconnected directoroleg122-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2334720, 279) != expected (2334720, 280) oleg122-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2334720, 279) != expected (2334720, 280) oleg122-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (2334720, 279) != expected (2334720, 280) pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 y inode 20101 (was in /ROOT/d30.sanity-lfsck/foo) Connect to /lost+found? yes Unconnected directory inode 20103 (was in /lost+found/#20101) Connect to /lost+found? yes Pass 3A: Memory used: 264k/0k (89k/176k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 264k/0k (87k/178k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 1865.67MB/s Pass 4: Checking reference counts Unattached inode 183 Connect to /lost+found? yes Inode 183 ref count is 2, should be 1. Fix? yes Unattached inode 184 Connect to /lost+found? yes Inode 184 ref count is 2, should be 1. Fix? yes Unattached inode 187 Connect to /lost+found? yes Inode 187 ref count is 2, should be 1. Fix? yes Unattached inode 195 Connect to /lost+found? yes Inode 195 ref count is 2, should be 1. Fix? yes Inode 20099 ref count is 1, should be 2. Fix? yes Inode 20103 ref count is 3, should be 2. Fix? yes Pass 4: Memory used: 264k/0k (70k/195k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 1MB, rate: 402.09MB/s Pass 5: Checking group summary information Pass 5: Memory used: 264k/0k (69k/196k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 1MB, rate: 353.23MB/s Update quota info for quota type 0? yes Update quota info for quota type 1? yes Update quota info for quota type 2? yes lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** 289 inodes used (0.72%, out of 40000) 6 non-contiguous files (2.1%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 1/0/0 12656 blocks used (50.62%, out of 25000) 0 bad blocks 1 large file 156 regular files 123 directories 0 character device files 0 block device files 0 fifos 4294967294 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 273 files Memory used: 264k/0k (68k/197k), time: 0.04/ 0.01/ 0.02 I/O read: 1MB, write: 1MB, rate: 27.94MB/s oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Trigger namespace LFSCK to recover backend orphans Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg122-client.virtnet: -o user_xattr,flock oleg122-server@tcp:/lustre /mnt/lustre File: '/mnt/lustre/d30.sanity-lfsck/foo/f0' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920875 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-16 11:15:20.000000000 -0400 Modify: 2024-04-16 11:15:20.000000000 -0400 Change: 2024-04-16 11:15:29.000000000 -0400 Birth: - total 12 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 . 144115205306056705 drwx------ 3 root root 4096 Apr 16 11:14 MDT0000 162129603815538689 drwx------ 2 root root 4096 Apr 16 11:14 MDT0001 d0 should become orphan under .lustre/lost+found/MDT0000/ total 16 144115205306056705 drwx------ 3 root root 4096 Apr 16 11:14 . 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 .. 144115272414920839 -rw-r--r-- 1 root root 6 Apr 16 11:13 [0x2000013a3:0x87:0x0]-O-0 144115272414920856 -rw-r--r-- 1 root root 0 Apr 16 11:14 [0x2000013a3:0x98:0x0]-O-0 144115272414920876 drwxr-xr-x 3 root root 4096 Apr 16 11:15 [0x2000013a3:0xac:0x0]-[0x2000013a3:0xaa:0x0]-D-0 File: '/mnt/lustre/.lustre/lost+found/MDT0000/[0x2000013a3:0xac:0x0]-[0x2000013a3:0xaa:0x0]-D-0/d1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115272414920878 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-16 11:15:21.000000000 -0400 Modify: 2024-04-16 11:15:21.000000000 -0400 Change: 2024-04-16 11:15:29.000000000 -0400 Birth: - File: '/mnt/lustre/.lustre/lost+found/MDT0000/[0x2000013a3:0xac:0x0]-[0x2000013a3:0xaa:0x0]-D-0/f1' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272414920877 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-16 11:15:21.000000000 -0400 Modify: 2024-04-16 11:15:21.000000000 -0400 Change: 2024-04-16 11:15:29.000000000 -0400 Birth: -