From 6e1fe5cacba430002a43df7d272e5f5737f9cc03 Mon Sep 17 00:00:00 2001 From: Sean Chittenden Date: Sat, 18 Oct 2003 00:48:15 +0000 Subject: [PATCH] Revamp the first section regarding obtaining a kernel dump and bring the instructions inline with reality (ex: savecore hasn't had the -N flag for a while). The verbage here should be sufficient for developers to be able to point users on current@ to these first two sections and have them extract something reasonably useful, esp now that -CURRENT no longer means FreeBSD 3.1. --- .../kerneldebug/chapter.sgml | 172 +++++++++++------- 1 file changed, 103 insertions(+), 69 deletions(-) diff --git a/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.sgml b/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.sgml index 5e7db29974..64494cd32f 100644 --- a/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.sgml +++ b/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.sgml @@ -9,94 +9,128 @@ Contributed by &a.paul; and &a.joerg; - - Debugging a Kernel Crash Dump with <command>gdb</command> + + Obtaining a Kernel Crash Dump - Here are some instructions for getting kernel debugging - working on a crash dump. They assume that you have enough swap - space for a crash dump. Typically you want to - specify one of the swap devices specified in - /etc/fstab. Dumps to non-swap devices, - tapes for example, are currently not supported. + When running a development kernel (eg: &os.current;), a + kernel under extreme conditions (eg: very high load averages, + tens of thousands of connections, exceedingly high number of + concurrent users, hundreds of &man.jail.8;s, etc.), or using a + new feature or device driver on &os.stable; (eg: + PAE), sometimes a kernel will panic. In the + event that it does, this chapter includes basic instructions for + extracting useful information out of a crash. - - Use the &man.dumpon.8; command to tell the kernel where to - save crash dumps. The dumpon program must - be called after the swap partition has been configured with - &man.swapon.8;. This is normally arranged by setting the - dumpdev variable in &man.rc.conf.5;. If - this variable is set, then the &man.savecore.8; program will - automatically be called on the first multi-user boot after the - crash. This program will save the kernel crash dump to the - directory specified in the rc.conf - dumpdir variable. The default directory - for crash dumps is /var/crash. + A system reboot is inevitable once a kernel panics. Once a + system is rebooted, the contents of a system's physical memory + (RAM) is lost, as well as any bits that are + on the swap device before the panic. To preserve the bits in + physical memory, the kernel makes use of the swap device as a + place to store the bits that are in physical memory that way + when the system reboots, a kernel image can be extracted and + debugging can take place. + + A swap device that has been configured as a dump + device still acts as a swap device. Dumps to non-swap devices, + tapes for example, are not supported at this time. A + swap device is synonymous with a swap + partition. + + To be able to extract a usable core, it is required that at + least one swap partition be large enough to hold all of the bits + in physical memory. When a kernel panics, before the system + reboots, the kernel is smart enough to check to see if a swap + device has been configured as a dump device. If there is a + valid dump device, the kernel dumps the contents of what is in + physical memory to the swap device (assuming the swap device is + configured as a dump device). + + + Configuring the Dump Device + + Before the kernel will dump the contents of its physical + memory to a dump device, a dump device must be configured. A + dump device is specified by using the &man.dumpon.8; command + to tell the kernel where to save kernel crash dumps. The + &man.dumpon.8; program must be called after the swap partition + has been configured with &man.swapon.8;. This is normally + handled by setting the dumpdev variable in + &man.rc.conf.5; to the path of the swap device. Alternatively, you can hard-code the dump device via the dump clause in the config line of your kernel configuration file. This approach is deprecated and should be used only if you want a crash dump from a kernel that crashes during booting. - - - In the following, the term gdb refers to - the debugger gdb run in kernel debug - mode. This can be accomplished by starting the - gdb with the option . In - kernel debug mode, gdb changes its prompt to - (kgdb). - + Check /etc/fstab or + &man.swapinfo.8; for a list of swap devices. + + Make sure the dumpdir + specified in &man.rc.conf.5; exists before a kernel + crash! + + &prompt.root; mkdir /var/crash + + + + + Extracting a Kernel Dump + + Once a dump has been written to a dump device, the dump + must be extracted before the swap device is mounted, + otherwise the dump will be corrupted. To extract a dump + from a dump device, use the &man.savecore.8; program. If + dumpdev has been set in &man.rc.conf.5;, + &man.savecore.8; will be called automatically on the first + multi-user boot after the crash and before the swap device + is mounted. The location of the extracted core is placed in + the &man.rc.conf.5; value dumpdir, by + default /var/crash. - If you are using FreeBSD 3 or earlier, you should make a stripped - copy of the debug kernel, rather than installing the large debug - kernel itself: - - &prompt.root; cp kernel kernel.debug -&prompt.root; strip -g kernel - - This stage is not necessary, but it is recommended. (In - FreeBSD 4 and later releases this step is performed automatically - at the end of the kernel make process.) - When the kernel has been stripped, either automatically or by - using the commands above, you may install it as usual by typing - make install. - - Note that older releases of FreeBSD (up to but not including - 3.1) used a.out kernels by default, which must have their symbol - tables permanently resident in physical memory. With the larger - symbol table in an unstripped debug kernel, this is wasteful. - Recent FreeBSD releases use ELF kernels where this is no longer a - problem. - - - If you are testing a new kernel, for example by typing the new - kernel's name at the boot prompt, but need to boot a different one in + If you are testing a new kernel but need to boot a different one in order to get your system up and running again, boot it only into single - user state using the flag at the boot prompt, and + user mode using the flag at the boot prompt, and then perform the following steps: &prompt.root; fsck -p -&prompt.root; mount -a -t ufs # so your filesystem for /var/crash is writable -&prompt.root; savecore -N /kernel.panicked /var/crash -&prompt.root; exit # ...to multi-user +&prompt.root; mount -a -t ufs # make sure /var/crash is writable +&prompt.root; savecore /var/crash /dev/ad0s1b +&prompt.root; exit # exit to multi-user - This instructs &man.savecore.8; to use another kernel for symbol - name extraction. It would otherwise default to the currently running - kernel and most likely not do anything at all since the crash dump and - the kernel symbols differ. + This instructs &man.savecore.8; to extract a kernel dump + from /dev/ad0s1b and place the contents in + /var/crash. Don't forget to make sure the + destination directory /var/crash has enough + space for the dump or to specify the correct path to your swap + device as it is likely different than + /dev/ad0s1b! - Now, after a crash dump, go to - /sys/compile/WHATEVER and run - gdb . From gdb do: + The recommended and certainly easiest way to automate + obtaining crash dumps is to use the dumpdev + variable in &man.rc.conf.5;. - symbol-file kernel.debug -exec-file /var/crash/kernel.0 -core-file /var/crash/vmcore.0 + + - and voila, you can debug the crash dump using the kernel sources just - like you can for any other program. + + Debugging a Kernel Crash Dump with <command>gdb</command> + + Once a dump has been obtained, getting useful information + out of the dump is relatively easy for simple problems. Before + launching into the internals of gdb to debug + the crash dump, locate the debug version of your kernel + (normally called kernel.debug) and the path + to the source files used to build your kernel (normally + /usr/obj/usr/src/sys/KERNCONF). With those + two pieces of info, let the debugging commence! + + &prompt.root; cd /usr/obj/usr/src/sys/KERNCONF +&prompt.root; gdb -k /boot/kernel/kernel.debug /var/crash/vmcore.0 + + and voila! You can debug the crash dump using the kernel + sources just like you can for any other program. Here is a script log of a gdb session illustrating the procedure. Long lines have been folded to improve