Debugging the Panic with crash
Time to figure out what went wrong and to find where in the source code the problem occurs.
To be able to start crash, we need a kernel with debugging symbols built in. Now, this kernel has to exactly match the production kernel that we crashed, and cannot be compressed. For Ubuntu systems, we are looking for the kernel ddeb.
We can add the ddeb repository to our system with:
$ sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse
EOF
if you get below error
Reading package lists... Done
E: The repository 'http://ddebs.ubuntu.com xenial-security Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
add [trusted=yes] for example
/# cat /etc/apt/sources.list.d/ddebs.list
deb [trusted=yes] http://ddebs.ubuntu.com/ xenial main restricted universe multiverse
deb [trusted=yes] http://ddebs.ubuntu.com/ xenial-security main restricted universe multiverse
deb [trusted=yes] http://ddebs.ubuntu.com/ xenial-updates main restricted universe multiverse
deb [trusted=yes] http://ddebs.ubuntu.com/ xenial-proposed main restricted universe multiverse
We can then import the GPG key for the repo, refresh package lists and install the vmlinux package with:-
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
$ sudo apt update
$ sudo apt install linux-image-$(uname -r)-dbgsym
Starting crash ....................
# crash <vmlinux> <dumpfile>
# crash /usr/lib/debug/boot/vmlinux-5.4.0-42-generic /var/crash/202008290918/dump.202008290918
crash 7.2.8
Copyright (C) 2002-2020 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: kernel relocated [556MB]: patching 114875 gdb minimal_symbol values
KERNEL: /usr/lib/debug/boot/vmlinux-5.4.0-42-generic
DUMPFILE: /var/crash/202008290918/dump.202008290918 [PARTIAL DUMP]
CPUS: 4
DATE: Sat Aug 29 09:16:55 2020
UPTIME: 00:04:40
LOAD AVERAGE: 1.97, 1.79, 0.80
TASKS: 504
NODENAME: user1-HP-Pavilion-dv4-Notebook-PC
RELEASE: 5.4.0-42-generic
VERSION: #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020
MACHINE: x86_64 (2494 Mhz)
MEMORY: 7.9 GB
PANIC: "Kernel panic - not syncing: sysrq triggered crash"
PID: 4488
COMMAND: "bash"
TASK: ffff89dec9895d00 [THREAD_INFO: ffff89dec9895d00]
CPU: 3
STATE: TASK_RUNNING (PANIC)
crash> log
[ 0.000000] microcode: microcode updated early to revision 0x21, date = 2019-02-13
[ 0.000000] Linux version 5.4.0-42-generic (buildd@lgw01-amd64-038) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 (Ubuntu 5.4.0-42.46-generic 5.4
.44)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-42-generic root=UUID=eb00fc0a-fab7-42a7-a272-7ba7b07dd94d ro quiet splash crashkernel=512M-:192M vt.handoff=7
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Hygon HygonGenuine
[ 0.000000] Centaur CentaurHauls
[ 0.000000] zhaoxin Shanghai
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000087fff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000000088000-0x00000000000bffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001fffffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000020000000-0x00000000201fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000020200000-0x0000000040003fff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000040004000-0x0000000040004fff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000040005000-0x00000000aa26efff] usable
Note that the crashed process is shown with ‘>’.
The " log " command brings up the contents of dmesg, for that particular session.
[ 132.749323] audit: type=1400 audit(1598706924.030:74): apparmor="DENIED" operation="open" profile="snap.snap-store.ubuntu-software" name="/var/lib/snapd/hostfs/usr/share/icons/Yaru/8x8/emblems/" pid=3997 comm="pool-org.gnome." requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 132.751313] audit: type=1400 audit(1598706924.030:75): apparmor="DENIED" operation="open" profile="snap.snap-store.ubuntu-software" name="/var/lib/snapd/hostfs/usr/share/icons/Yaru/icon-theme.cache" pid=3997 comm="pool-org.gnome." requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 223.805844] sysrq: Trigger a crash
[ 223.805853] Kernel panic - not syncing: sysrq triggered crash
[ 223.805861] CPU: 3 PID: 4488 Comm: bash Kdump: loaded Tainted: G OE 5.4.0-42-generic #46-Ubuntu
[ 223.805865] Hardware name: Hewlett-Packard HP Pavilion dv4 Notebook PC/1836, BIOS F.22 09/05/2012
[ 223.805868] Call Trace:
[ 223.805885] dump_stack+0x6d/0x9a
[ 223.805895] panic+0x101/0x2e3
[ 223.805903] sysrq_handle_crash+0x15/0x20
[ 223.805908] __handle_sysrq.cold+0x48/0x107
[ 223.805914] write_sysrq_trigger+0x28/0x40
[ 223.805920] proc_reg_write+0x43/0x70
[ 223.805926] __vfs_write+0x1b/0x40
[ 223.805931] vfs_write+0xb9/0x1a0
[ 223.805937] ksys_write+0x67/0xe0
[ 223.805942] __x64_sys_write+0x1a/0x20
[ 223.805950] do_syscall_64+0x57/0x190
[ 223.805958] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 223.805965] RIP: 0033:0x7f1d2cb70057
[ 223.805971] Code: 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[ 223.805976] RSP: 002b:00007ffc65c9e978 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 223.805982] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f1d2cb70057
[ 223.805986] RDX: 0000000000000002 RSI: 0000558c871b6900 RDI: 0000000000000001
[ 223.805989] RBP: 0000558c871b6900 R08: 000000000000000a R09: 0000000000000001
[ 223.805993] R10: 0000558c851b2017 R11: 0000000000000246 R12: 0000000000000002
[ 223.805997] R13: 00007f1d2cc4b6a0 R14: 00007f1d2cc4c4a0 R15: 00007f1d2cc4b8a0
for more complex bugs, the bt command is interesting.
bt normally shows the backtrace of the task which caused the system crash, but you can select other tasks and fetch their backtraces as well.
The output for bt is:
crash> bt
PID: 4488 TASK: ffff89dec9895d00 CPU: 3 COMMAND: "bash"
#0 [ffffaaed81ee7c68] machine_kexec at ffffffffa3c700e3
#1 [ffffaaed81ee7cc8] __crash_kexec at ffffffffa3d58cd2
#2 [ffffaaed81ee7d98] panic at ffffffffa3ca1962
#3 [ffffaaed81ee7e18] sysrq_handle_crash at ffffffffa4290d15
#4 [ffffaaed81ee7e28] __handle_sysrq.cold at ffffffffa42916a8
#5 [ffffaaed81ee7e60] write_sysrq_trigger at ffffffffa4291548
#6 [ffffaaed81ee7e78] proc_reg_write at ffffffffa3f715a3
#7 [ffffaaed81ee7e98] __vfs_write at ffffffffa3edab1b
#8 [ffffaaed81ee7ea8] vfs_write at ffffffffa3edb829
#9 [ffffaaed81ee7ee0] ksys_write at ffffffffa3edddb7
#10 [ffffaaed81ee7f20] __x64_sys_write at ffffffffa3edde4a
#11 [ffffaaed81ee7f30] do_syscall_64 at ffffffffa3c044c7
#12 [ffffaaed81ee7f50] entry_SYSCALL_64_after_hwframe at ffffffffa480008c
RIP: 00007f1d2cb70057 RSP: 00007ffc65c9e978 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f1d2cb70057
RDX: 0000000000000002 RSI: 0000558c871b6900 RDI: 0000000000000001
RBP: 0000558c871b6900 R8: 000000000000000a R9: 0000000000000001
R10: 0000558c851b2017 R11: 0000000000000246 R12: 0000000000000002
R13: 00007f1d2cc4b6a0 R14: 00007f1d2cc4c4a0 R15: 00007f1d2cc4b8a0
ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b
crash>
No comments:
Post a Comment