************************ crashinfo ************************* /exports/testreports/42123/testresults/sanity-pfl-zfs-centos7_x86_64-centos7_x86_64/oleg254-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.waZWH/vmlinux [TAINTED] DUMPFILE: /exports/testreports/42123/testresults/sanity-pfl-zfs-centos7_x86_64-centos7_x86_64/oleg254-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Fri Apr 19 03:12:02 EDT 2024 UPTIME: 01:02:55 LOAD AVERAGE: 0.00, 0.01, 0.05 TASKS: 392 NODENAME: oleg254-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 22 last 5s 63 last 60s 75 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 388 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 7666 z_null_iss 0 ms (no user stack) 7742 mmp 0 ms (no user stack) 6214 z_null_int 0 ms (no user stack) 7667 z_null_int 0 ms (no user stack) 6283 mmp 0 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 714483 2.7 GB 74% of TOTAL MEM USED 240584 939.8 MB 25% of TOTAL MEM SHARED 8717 34.1 MB 0% of TOTAL MEM BUFFERS 5052 19.7 MB 0% of TOTAL MEM CACHED 65485 255.8 MB 6% of TOTAL MEM SLAB 25197 98.4 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 58575 228.8 MB 7% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 5 LISTEN 3 NAGLE disabled (TCP_NODELAY): 4 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 3768.4 eth0 n/a 0.4 RSS_TOTAL=53184 pages, %mem= 0.9 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff8801376688c0 ffff88012a340000 sysfs sysfs /sys ffff880137668a80 ffff880139944000 proc proc /proc ffff880137668c40 ffff880137678000 devtmpfs devtmpfs /dev ffff880137668e00 ffff8800b664f800 securityfs securityfs /sys/kernel/security ffff880137668fc0 ffff88012a340800 tmpfs tmpfs /dev/shm ffff880137669180 ffff880136b71000 devpts devpts /dev/pts ffff880137669340 ffff88012a341000 tmpfs tmpfs /run ffff880138ccac40 ffff8800b663e800 tmpfs tmpfs /sys/fs/cgroup ffff880138ccae00 ffff8800b663f000 cgroup cgroup /sys/fs/cgroup/systemd ffff880138ccafc0 ffff8800b663f800 pstore pstore /sys/fs/pstore ffff88012b2cc1c0 ffff8800b54d9000 cgroup cgroup /sys/fs/cgroup/devices ffff88012b2cc380 ffff8800b54d9800 cgroup cgroup /sys/fs/cgroup/pids ffff88012b2cc540 ffff8800b54da000 cgroup cgroup /sys/fs/cgroup/memory ffff88012b2cc700 ffff8800b54da800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012b2cc8c0 ffff8800b54db000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012b2cca80 ffff8800b54db800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012b2ccc40 ffff8800b54dc000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012b2cce00 ffff8800b54dc800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012b2ccfc0 ffff8800b54dd000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012b2cd180 ffff8800b54dd800 cgroup cgroup /sys/fs/cgroup/cpuset ffff880138ccb180 ffff880129c59800 configfs configfs /sys/kernel/config ffff88012b298c40 ffff8800b4982000 ext4 /dev/nbd0 / ffff88012b298e00 ffff8800b5247800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012b2cdc00 ffff8800b4825000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880138ccb340 ffff880136b72800 mqueue mqueue /dev/mqueue ffff88012b2cddc0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b425c700 ffff8800b491b000 hugetlbfs hugetlbfs /dev/hugepages ffff88012b2cda40 ffff8800b4826000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b425c8c0 ffff8800b491e800 ramfs none /mnt ffff88012b299180 ffff88012a345000 squashfs /dev/vda /home/green/git/lustre-release ffff8800b425cc40 ffff8800b491c000 tmpfs none /var/lib/stateless/writable ffff8800b425ce00 ffff8800b491c000 tmpfs none /var/cache/man ffff88012b299340 ffff8800b491c000 tmpfs none /var/log ffff88012b299500 ffff8800b491c000 tmpfs none /var/lib/dbus ffff88012b2996c0 ffff8800b491c000 tmpfs none /tmp ffff88012b299880 ffff8800b491c000 tmpfs none /var/lib/dhclient ffff8800b425cfc0 ffff8800b491c000 tmpfs none /var/tmp ffff88012b299a40 ffff8800b491c000 tmpfs none /var/lib/NetworkManager ffff88012b299c00 ffff8800b491c000 tmpfs none /var/lib/systemd/random-seed ffff8800b425d180 ffff8800b491c000 tmpfs none /var/spool ffff880138ccb500 ffff8800b491c000 tmpfs none /var/lib/nfs ffff880138ccb6c0 ffff8800b491c000 tmpfs none /var/lib/gssproxy ffff8800b425d340 ffff8800b491c000 tmpfs none /var/lib/logrotate ffff8800b425d500 ffff8800b491c000 tmpfs none /etc ffff88012b299dc0 ffff8800b491c000 tmpfs none /var/lib/rsyslog ffff88012b2988c0 ffff8800b491c000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b3048000 ffff8800b4821000 nfs4 192.168.200.253:/exports/state/oleg254-server.virtnet /var/lib/stateless/state ffff8800b3048540 ffff8800b4821000 nfs4 192.168.200.253:/exports/state/oleg254-server.virtnet /boot ffff88012b2cd880 ffff8800b4821000 nfs4 192.168.200.253:/exports/state/oleg254-server.virtnet /etc/etc/kdump.conf ffff8800b3048700 ffff8800b5247800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b3049a40 ffff88012a345000 squashfs /dev/vda /usr/sbin/mount.lustre ffff880137669c00 ffff8800b2217800 lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 ffff880099cf0c40 ffff88012d89a800 lustre lustre-ost1/ost1 /mnt/lustre-ost1 ffff8800b3049500 ffff880093aa7800 lustre lustre-ost2/ost2 /mnt/lustre-ost2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 162.821273] Lustre: DEBUG MARKER: == sanity-pfl test 0a: Create full components file, no reused OSTs ========================================================== 02:11:47 (1713507107) [ 163.786906] Lustre: DEBUG MARKER: sanity-pfl test_0a: @@@@@@ FAIL: 1 == 1 [ 171.745504] Lustre: DEBUG MARKER: == sanity-pfl test 0b: Verify comp stripe count limits === 02:11:56 (1713507116) [ 212.007507] Lustre: mdt00_003: service thread pid 11396 was inactive for 40.019 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 212.018491] Pid: 11396, comm: mdt00_003 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 212.026497] Call Trace: [ 212.027198] [<0>] osp_precreate_reserve+0x59f/0xaa0 [osp] [ 212.031485] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [ 212.034242] [<0>] lod_sub_declare_create+0xe6/0x280 [lod] [ 212.036343] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [ 212.037702] [<0>] lod_ost_alloc_rr.constprop.22+0xb04/0x1200 [lod] [ 212.039974] [<0>] lod_qos_prep_create+0x121a/0x1ab0 [lod] [ 212.041948] [<0>] lod_prepare_create+0x23b/0x320 [lod] [ 212.044065] [<0>] lod_declare_striped_create+0xf8/0xa50 [lod] [ 212.046166] [<0>] lod_declare_xattr_set+0x139/0x20c0 [lod] [ 212.047520] [<0>] mdd_create_data+0x462/0x720 [mdd] [ 212.048767] [<0>] mdt_mfd_open+0xccc/0xee0 [mdt] [ 212.049916] [<0>] mdt_finish_open+0x803/0x9e0 [mdt] [ 212.051735] [<0>] mdt_open_by_fid_lock+0x55b/0xb90 [mdt] [ 212.057303] [<0>] mdt_reint_open+0x778/0x2b30 [mdt] [ 212.060859] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 212.063954] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 212.067246] [<0>] mdt_intent_open+0x93/0x480 [mdt] [ 212.071204] [<0>] mdt_intent_opc+0x1dd/0xc10 [mdt] [ 212.079867] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [ 212.085735] [<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc] [ 212.093925] [<0>] ldlm_handle_enqueue0+0x8c6/0x1780 [ptlrpc] [ 212.101079] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [ 212.106635] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 212.111102] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 212.117260] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 212.121856] [<0>] kthread+0xe4/0xf0 [ 212.124036] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 212.128148] [<0>] 0xfffffffffffffffe [ 767.271510] Lustre: 6509:0:(service.c:1437:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff88008eed7200 x1796742317090688/t0(0) o101->e13bb98d-5b8c-4ab0-85bd-349123982f8b@192.168.202.54@tcp:487/0 lens 760/3536 e 24 to 0 dl 1713507717 ref 2 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 774.012165] Lustre: lustre-MDT0000: Client e13bb98d-5b8c-4ab0-85bd-349123982f8b (at 192.168.202.54@tcp) reconnecting [ 1375.030790] Lustre: lustre-MDT0000: Client e13bb98d-5b8c-4ab0-85bd-349123982f8b (at 192.168.202.54@tcp) reconnecting [ 1976.054074] Lustre: lustre-MDT0000: Client e13bb98d-5b8c-4ab0-85bd-349123982f8b (at 192.168.202.54@tcp) reconnecting [ 2577.069389] Lustre: lustre-MDT0000: Client e13bb98d-5b8c-4ab0-85bd-349123982f8b (at 192.168.202.54@tcp) reconnecting [ 3178.087439] Lustre: lustre-MDT0000: Client e13bb98d-5b8c-4ab0-85bd-349123982f8b (at 192.168.202.54@tcp) reconnecting ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 12.24s (real) 6.95s (CPU), Child processes: 5.26s