kdump is a tool which captures crash dumps from the memory of a running crashed system. The architecture is slightly interesting, since when the production kernel crashes, another, separate kernel is booted via kexec, into a memory space reserved by the production kernel. The freshly booted separate kernel, also known as the crash kernel, then captures the state of the crashed system, and writes it to disk.
This all works because of kexec. kexec is a fastboot tool implemented in the kernel. It allows other kernels to be booted from an already running kernel, skipping the need to go through standard BIOS routines. This, coupled with the fact that the crash kernel is loaded into reserved space allocated by the production kernel, means that memory is intact and is not overwritten by a reboot. This is how reliability and integrity of crash dumps are assured.
Once the crash dump is written to disk, or even sent over the network to a remote host, the system reboots so uptime can be restored. The crash dump can then be analysed later on.
Configuring kdump
Kernel Configuration
If we read the kdump Documentation, kdump requires some kernel features to be configured and compiled into the kernel.
They are:
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PROC_VMCORE=y
CONFIG_DEBUG_INFO=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_KEXEC enables the syscall required for kexec to function, and is necessary for being able to boot into the crash kernel.
CONFIG_CRASH_DUMP and CONFIG_PROC_VMCORE enables the crashed kernel to be dumped from memory, and exported to a ELF file.
CONFIG_DEBUG_INFO builds the kernel with debugging symbols, and produces a vmlinux file which can be used for analysis of crash dumps. The production kernel runs a kernel stripped of debugging symbols for performance, so it is very important to match packages of production kernels and debug kernels.
CONFIG_MAGIC_SYSRQ is necessary to be able to use SYSRQ features, such as flushing buffers on kernel panic, and to be able to trigger crashes manually.
CONFIG_RELOCATABLE is set since our kernel is relocatable in memory, so we must also set CONFIG_PHYSICAL_START as the deterministic address in which we can place the crash kernel in memory. Now, 0x1000000 is at 16mb in physical memory, and is the default set in Cosmic Cuttlefish’s kernel.
The nice thing is, all of these features are enabled by default on Ubuntu production kernels, so we don’t need to compile our own kernel today. You can verify that these features are enabled by looking at the config files in /boot:
root@user1-HP-Pavilion-dv4-Notebook-PC:/home/user1# grep -i crash /boot/config-5.4.0-42-generic
CONFIG_CRASH_DUMP=y
CONFIG_CRASH_CORE=y
Installing Packages
We can install all the required kdump packages with the command:
sudo apt install linux-crashdump
This will install crash, kdump-tools, kexec-tools, makedumpfile and itself.
Should kexec-tools handle reboots (sysvinit only)? answer YES
Should kdump-tools be enabled by default? asnwer YES
We should probably make sure we have all the necessary kernel packages around as well, such as headers and tools, so run:
$ sudo apt install linux-image-generic linux-headers-generic linux-tools-generic
Setting up kdump-tools
We need to tell the kernel where the crash kernel will be loaded in memory, and how much space it has. This happens by appending a crashkernel=... to the kernel command line. This has been done for us when we installed linux-crashdump, and we need to reboot the system so that the changes actually take effect. So go ahead and restart.
After restarting, we can see run kdump-config show, to see that all is well:
root@user1-HP-Pavilion-dv4-Notebook-PC:/home/user1# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x8a000000
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-42-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-42-generic
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-42-generic root=UUID=eb00fc0a-fab7-42a7-a272-7ba7b07dd94d ro quiet splash vt.handoff=7 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
A few interesting things to note:
Core dumps will be stored in /var/crash
The crash kernel vmlinuz is located at /var/lib/kdump/vmlinuz
The crash kernel initial filesystem is located at /var/lib/kdump/initrd.img
To see the updated kernel command line, we can run cat /proc/cmdline:
root@user1-HP-Pavilion-dv4-Notebook-PC:/home/user1# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.4.0-42-generic root=UUID=eb00fc0a-fab7-42a7-a272-7ba7b07dd94d ro quiet splash crashkernel=512M-:192M vt.handoff=7
$ cat /proc/sys/kernel/sysrq
176
We will have to look at /etc/sysctl.d/10-magic-sysrq.conf to see what 176 means.
$ sudo sysctl -w kernel.sysrq=1
Right. We can trigger a system crash by passing the ‘c’ character to /proc/sysrq-trigger. This has to be done as root:
$ sudo -s
# echo c > /proc/sysrq-trigger
No comments:
Post a Comment