************************ crashinfo ************************* /exports/testreports/42123/testresults/sanity-pfl-special1-ldiskfs-centos7_x86_64-centos7_x86_64/oleg341-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.uLGHz/vmlinux [TAINTED] DUMPFILE: /exports/testreports/42123/testresults/sanity-pfl-special1-ldiskfs-centos7_x86_64-centos7_x86_64/oleg341-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Fri Apr 19 03:12:20 EDT 2024 UPTIME: 01:03:17 LOAD AVERAGE: 0.00, 0.01, 0.05 TASKS: 219 NODENAME: oleg341-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 15 last 5s 43 last 60s 58 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 215 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 3857 dbuf_evict 0 ms (no user stack) 934 in:imjournal 0 ms /usr/sbin/rsyslogd -n 7389 ldlm_bl_03 0 ms (no user stack) 24 kworker/2:0 0 ms (no user stack) 50 kworker/0:1 73 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 757284 2.9 GB 79% of TOTAL MEM USED 197783 772.6 MB 20% of TOTAL MEM SHARED 10246 40 MB 1% of TOTAL MEM BUFFERS 7328 28.6 MB 0% of TOTAL MEM CACHED 65452 255.7 MB 6% of TOTAL MEM SLAB 14820 57.9 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 58060 226.8 MB 7% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 5 LISTEN 3 NAGLE disabled (TCP_NODELAY): 4 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 3788.7 eth0 n/a 2.7 RSS_TOTAL=57292 pages, %mem= 0.9 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880137668380 ffff8800b5288800 sysfs sysfs /sys ffff880137668540 ffff880139944000 proc proc /proc ffff880137668700 ffff880137678000 devtmpfs devtmpfs /dev ffff8801376688c0 ffff8800b7ea7000 securityfs securityfs /sys/kernel/security ffff880137668a80 ffff8800b5289000 tmpfs tmpfs /dev/shm ffff880137668c40 ffff880137321000 devpts devpts /dev/pts ffff880137668e00 ffff8800b5289800 tmpfs tmpfs /run ffff880137668fc0 ffff8800b528a000 tmpfs tmpfs /sys/fs/cgroup ffff880137669180 ffff8800b528a800 cgroup cgroup /sys/fs/cgroup/systemd ffff880137669340 ffff8800b528b000 pstore pstore /sys/fs/pstore ffff880137669500 ffff8800b528d000 cgroup cgroup /sys/fs/cgroup/perf_event ffff8801376696c0 ffff8800b528c800 cgroup cgroup /sys/fs/cgroup/devices ffff880137669880 ffff8800b528c000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff880137669a40 ffff8800b528b800 cgroup cgroup /sys/fs/cgroup/blkio ffff880137669c00 ffff8800b528d800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff880137669dc0 ffff8800b528e000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a330000 ffff8800b528e800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a3301c0 ffff8800b528f000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a330380 ffff8800b528f800 cgroup cgroup /sys/fs/cgroup/memory ffff880138ccac40 ffff8800b638d800 cgroup cgroup /sys/fs/cgroup/pids ffff8801370d6c40 ffff8800b4cc6000 configfs configfs /sys/kernel/config ffff8801370d6e00 ffff8800b4cc3800 ext4 /dev/nbd0 / ffff880138ccafc0 ffff88012a316800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012b2aec40 ffff8800b4cc1000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8801370d6fc0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8801370d7180 ffff880137009800 hugetlbfs hugetlbfs /dev/hugepages ffff88012a331a40 ffff8801370dc800 mqueue mqueue /dev/mqueue ffff88012a331c00 ffff880129e24000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8801370d7340 ffff88013700b800 ramfs none /mnt ffff8800b2826000 ffff8800b3eb6000 squashfs /dev/vda /home/green/git/lustre-release ffff88012b2aee00 ffff8800b4cc6800 tmpfs none /var/lib/stateless/writable ffff8801370d7500 ffff8800b4cc6800 tmpfs none /var/cache/man ffff8800b28261c0 ffff8800b4cc6800 tmpfs none /var/log ffff88012b2aefc0 ffff8800b4cc6800 tmpfs none /var/lib/dbus ffff8801370d76c0 ffff8800b4cc6800 tmpfs none /tmp ffff88012b2af180 ffff8800b4cc6800 tmpfs none /var/lib/dhclient ffff88012b2af340 ffff8800b4cc6800 tmpfs none /var/tmp ffff88012b2af500 ffff8800b4cc6800 tmpfs none /var/lib/NetworkManager ffff88012b2af6c0 ffff8800b4cc6800 tmpfs none /var/lib/systemd/random-seed ffff88012b2af880 ffff8800b4cc6800 tmpfs none /var/spool ffff88012b2afa40 ffff8800b4cc6800 tmpfs none /var/lib/nfs ffff8800b2826380 ffff8800b4cc6800 tmpfs none /var/lib/gssproxy ffff88012b2afc00 ffff8800b4cc6800 tmpfs none /var/lib/logrotate ffff8800b2826540 ffff8800b4cc6800 tmpfs none /etc ffff88012b2afdc0 ffff8800b4cc6800 tmpfs none /var/lib/rsyslog ffff88012b2aea80 ffff8800b4cc6800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8801370d7880 ffff8800b3eb4800 nfs4 192.168.200.253:/exports/state/oleg341-server.virtnet /var/lib/stateless/state ffff88012b2ae8c0 ffff8800b3eb4800 nfs4 192.168.200.253:/exports/state/oleg341-server.virtnet /boot ffff88012b2ae700 ffff8800b3eb4800 nfs4 192.168.200.253:/exports/state/oleg341-server.virtnet /etc/etc/kdump.conf ffff8801370d7c00 ffff88012a316800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b21db180 ffff8800b3eb6000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b2827180 ffff8800b4ee1000 lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 ffff880138ccb880 ffff8800783ff000 lustre /dev/mapper/ost1_flakey /mnt/lustre-ost1 ffff880138ccbc00 ffff88007ba54800 lustre /dev/mapper/ost2_flakey /mnt/lustre-ost2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 169.376504] random: crng init done [ 171.819965] Lustre: DEBUG MARKER: excepting tests: [ 174.883445] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing check_config_client /mnt/lustre [ 184.882617] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 186.799827] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 189.074630] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 191.485649] Lustre: DEBUG MARKER: == sanity-pfl test 1c: Test overstriping w/max stripe count ========================================================== 02:12:12 (1713507132) [ 232.183317] Lustre: mdt00_003: service thread pid 7396 was inactive for 40.084 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 232.189422] Pid: 7396, comm: mdt00_003 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 232.195168] Call Trace: [ 232.196307] [<0>] osp_precreate_reserve+0x59f/0xaa0 [osp] [ 232.199774] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [ 232.203562] [<0>] lod_sub_declare_create+0xe6/0x280 [lod] [ 232.206303] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [ 232.210350] [<0>] lod_ost_alloc_rr.constprop.22+0xb04/0x1200 [lod] [ 232.215899] [<0>] lod_qos_prep_create+0x121a/0x1ab0 [lod] [ 232.218283] [<0>] lod_declare_instantiate_components+0xa7/0x1e0 [lod] [ 232.231429] [<0>] lod_declare_update_plain+0x67b/0x970 [lod] [ 232.235281] [<0>] lod_declare_layout_change+0x69e/0xb90 [lod] [ 232.237056] [<0>] mdd_declare_layout_change+0x4b/0x100 [mdd] [ 232.240302] [<0>] mdd_layout_change+0xd91/0x1b90 [mdd] [ 232.243449] [<0>] mdt_layout_change+0x2ff/0x490 [mdt] [ 232.245900] [<0>] mdt_intent_layout+0x910/0xe60 [mdt] [ 232.250136] [<0>] mdt_intent_opc+0x1dd/0xc10 [mdt] [ 232.251650] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [ 232.255436] [<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc] [ 232.257375] [<0>] ldlm_handle_enqueue0+0x8c6/0x1780 [ptlrpc] [ 232.258963] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [ 232.260102] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 232.261842] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 232.263795] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 232.266722] [<0>] kthread+0xe4/0xf0 [ 232.272348] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 232.274413] [<0>] 0xfffffffffffffffe [ 787.191180] Lustre: 6160:0:(service.c:1437:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff88007cd62f80 x1796742322331776/t0(0) o101->296ef4ab-fc9c-4321-bd59-4689d75f00e7@192.168.203.41@tcp:503/0 lens 376/904 e 24 to 0 dl 1713507733 ref 2 fl Interpret:/0/0 rc 0/0 job:'dd.0' [ 794.133081] Lustre: lustre-MDT0000: Client 296ef4ab-fc9c-4321-bd59-4689d75f00e7 (at 192.168.203.41@tcp) reconnecting [ 1395.163009] Lustre: lustre-MDT0000: Client 296ef4ab-fc9c-4321-bd59-4689d75f00e7 (at 192.168.203.41@tcp) reconnecting [ 1996.182459] Lustre: lustre-MDT0000: Client 296ef4ab-fc9c-4321-bd59-4689d75f00e7 (at 192.168.203.41@tcp) reconnecting [ 2597.212648] Lustre: lustre-MDT0000: Client 296ef4ab-fc9c-4321-bd59-4689d75f00e7 (at 192.168.203.41@tcp) reconnecting [ 3198.250209] Lustre: lustre-MDT0000: Client 296ef4ab-fc9c-4321-bd59-4689d75f00e7 (at 192.168.203.41@tcp) reconnecting ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.54s (real) 6.12s (CPU), Child processes: 5.40s