829 lines
37 KiB
Text
829 lines
37 KiB
Text
<!--
|
|
The FreeBSD Documentation Project
|
|
|
|
$FreeBSD$
|
|
-->
|
|
|
|
<chapter id="kerneldebug">
|
|
<chapterinfo>
|
|
<authorgroup>
|
|
<author>
|
|
<firstname>Paul</firstname>
|
|
<surname>Richards</surname>
|
|
<contrib>Contributed by </contrib>
|
|
</author>
|
|
<author>
|
|
<firstname>Jörg</firstname>
|
|
<surname>Wunsch</surname>
|
|
</author>
|
|
</authorgroup>
|
|
</chapterinfo>
|
|
|
|
<title>Kernel Debugging</title>
|
|
|
|
<sect1 id="kerneldebug-obtain">
|
|
<title>Obtaining a Kernel Crash Dump</title>
|
|
|
|
<para>When running a development kernel (eg: &os.current;), such as a
|
|
kernel under extreme conditions (eg: very high load averages,
|
|
tens of thousands of connections, exceedingly high number of
|
|
concurrent users, hundreds of &man.jail.8;s, etc.), or using a
|
|
new feature or device driver on &os.stable; (eg:
|
|
<acronym>PAE</acronym>), sometimes a kernel will panic. In the
|
|
event that it does, this chapter will demonstrate how to extract
|
|
useful information out of a crash.</para>
|
|
|
|
<para>A system reboot is inevitable once a kernel panics. Once a
|
|
system is rebooted, the contents of a system's physical memory
|
|
(<acronym>RAM</acronym>) is lost, as well as any bits that are
|
|
on the swap device before the panic. To preserve the bits in
|
|
physical memory, the kernel makes use of the swap device as a
|
|
temporary place to store the bits that are in RAM across a
|
|
reboot after a crash. In doing this, when &os; boots after a
|
|
crash, a kernel image can now be extracted and debugging can
|
|
take place.</para>
|
|
|
|
<note><para>A swap device that has been configured as a dump
|
|
device still acts as a swap device. Dumps to non-swap devices
|
|
(such as tapes or CDRWs, for example) are not supported at this time. A
|
|
<quote>swap device</quote> is synonymous with a <quote>swap
|
|
partition.</quote></para></note>
|
|
|
|
<para>To be able to extract a usable core, it is required that at
|
|
least one swap partition be large enough to hold all of the bits
|
|
in physical memory. When a kernel panics, before the system
|
|
reboots, the kernel is smart enough to check to see if a swap
|
|
device has been configured as a dump device. If there is a
|
|
valid dump device, the kernel dumps the contents of what is in
|
|
physical memory to the swap device.</para>
|
|
|
|
<sect2 id="config-dumpdev">
|
|
<title>Configuring the Dump Device</title>
|
|
|
|
<para>Before the kernel will dump the contents of its physical
|
|
memory to a dump device, a dump device must be configured. A
|
|
dump device is specified by using the &man.dumpon.8; command
|
|
to tell the kernel where to save kernel crash dumps. The
|
|
&man.dumpon.8; program must be called after the swap partition
|
|
has been configured with &man.swapon.8;. This is normally
|
|
handled by setting the <varname>dumpdev</varname> variable in
|
|
&man.rc.conf.5; to the path of the swap device (the
|
|
recommended way to extract a kernel dump).</para>
|
|
|
|
<para>Alternatively, the dump device can be hard-coded via the
|
|
<literal>dump</literal> clause in the &man.config.5; line of
|
|
a kernel configuration file. This approach is deprecated and should
|
|
be used only if a kernel is crashing before &man.dumpon.8; can be executed.</para>
|
|
|
|
<tip><para>Check <filename>/etc/fstab</filename> or
|
|
&man.swapinfo.8; for a list of swap devices.</para></tip>
|
|
|
|
<important><para>Make sure the <varname>dumpdir</varname>
|
|
specified in &man.rc.conf.5; exists before a kernel
|
|
crash!</para>
|
|
|
|
<screen>&prompt.root; <userinput>mkdir /var/crash</userinput>
|
|
&prompt.root; <userinput>chmod 700 /var/crash</userinput></screen>
|
|
|
|
<para>Also, remember that the contents of
|
|
<filename>/var/crash</filename> is sensitive and very likely
|
|
contains confidential information such as passwords.</para>
|
|
</important>
|
|
</sect2>
|
|
|
|
<sect2 id="extract-dump">
|
|
<title>Extracting a Kernel Dump</title>
|
|
|
|
<para>Once a dump has been written to a dump device, the dump
|
|
must be extracted before the swap device is mounted.
|
|
To extract a dump
|
|
from a dump device, use the &man.savecore.8; program. If
|
|
<varname>dumpdev</varname> has been set in &man.rc.conf.5;,
|
|
&man.savecore.8; will be called automatically on the first
|
|
multi-user boot after the crash and before the swap device
|
|
is mounted. The location of the extracted core is placed in
|
|
the &man.rc.conf.5; value <varname>dumpdir</varname>, by
|
|
default <filename>/var/crash</filename> and will be named
|
|
<filename>vmcore.0</filename>.</para>
|
|
|
|
<para>In the event that there is already a file called
|
|
<filename>vmcore.0</filename> in
|
|
<filename>/var/crash</filename> (or whatever
|
|
<varname>dumpdev</varname> is set to), the kernel will
|
|
increment the trailing number for every crash to avoid
|
|
overwriting an existing <filename>vmcore</filename> (eg:
|
|
<filename>vmcore.1</filename>). While debugging, it is
|
|
highly likely that you will want to use the highest version
|
|
<filename>vmcore</filename> in
|
|
<filename>/var/crash</filename> when searching for the right
|
|
<filename>vmcore</filename>.</para>
|
|
|
|
<tip>
|
|
<para>If you are testing a new kernel but need to boot a different one in
|
|
order to get your system up and running again, boot it only into single
|
|
user mode using the <option>-s</option> flag at the boot prompt, and
|
|
then perform the following steps:</para>
|
|
|
|
<screen>&prompt.root; <userinput>fsck -p</userinput>
|
|
&prompt.root; <userinput>mount -a -t ufs</userinput> # make sure /var/crash is writable
|
|
&prompt.root; <userinput>savecore /var/crash /dev/ad0s1b</userinput>
|
|
&prompt.root; <userinput>exit</userinput> # exit to multi-user</screen>
|
|
|
|
<para>This instructs &man.savecore.8; to extract a kernel dump
|
|
from <filename>/dev/ad0s1b</filename> and place the contents in
|
|
<filename>/var/crash</filename>. Don't forget to make sure the
|
|
destination directory <filename>/var/crash</filename> has enough
|
|
space for the dump. Also, don't forget to specify the correct path to your swap
|
|
device as it is likely different than
|
|
<filename>/dev/ad0s1b</filename>!</para></tip>
|
|
|
|
<para>The recommended, and certainly the easiest way to automate
|
|
obtaining crash dumps is to use the <varname>dumpdev</varname>
|
|
variable in &man.rc.conf.5;.</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<sect1 id="kerneldebug-gdb">
|
|
<title>Debugging a Kernel Crash Dump with <command>gdb</command></title>
|
|
|
|
<para>Once a dump has been obtained, getting useful information
|
|
out of the dump is relatively easy for simple problems. Before
|
|
launching into the internals of <command>gdb</command> to debug
|
|
the crash dump, locate the debug version of your kernel
|
|
(normally called <filename>kernel.debug</filename>) and the path
|
|
to the source files used to build your kernel (normally
|
|
<filename>/usr/obj/usr/src/sys/KERNCONF</filename>, where
|
|
<filename>KERNCONF</filename> is the <varname>ident</varname>
|
|
specified in a kernel &man.config.5;). With those two pieces of
|
|
info, let the debugging commence!</para>
|
|
|
|
<para>To enter into the debugger and begin getting information
|
|
from the dump, the following steps are required at a minimum:
|
|
|
|
<screen>&prompt.root; <userinput>cd /usr/obj/usr/src/sys/KERNCONF</userinput>
|
|
&prompt.root; <userinput>gdb -k /boot/kernel/kernel.debug /var/crash/vmcore.0</userinput></screen>
|
|
|
|
You can debug the crash dump using the kernel sources just like
|
|
you can for any other program.</para>
|
|
|
|
<para>This first dump is from a 5.2-BETA kernel and the crash
|
|
comes from deep within the kernel. The ouput below has been
|
|
modified to include line numbers on the left. This first trace
|
|
inspects the instruction pointer and obtains a back trace. The
|
|
address that is used on line 41 for the <command>list</command>
|
|
command is the instruction pointer and can be found on line line
|
|
17. Most developers will request having at least this
|
|
information sent to them if you are unable to debug the problem
|
|
yourself. If, however, you do solve the problem, make sure that
|
|
your patch winds its way into the source tree via a problem
|
|
report, mailing lists, or by being able to commit it!
|
|
|
|
<screen> 1:&prompt.root; <userinput>cd /usr/obj/usr/src/sys/KERNCONF</userinput>
|
|
2:&prompt.root; <userinput>gdb -k kernel.debug /var/crash/vmcore.0</userinput>
|
|
3:GNU gdb 5.2.1 (FreeBSD)
|
|
4:Copyright 2002 Free Software Foundation, Inc.
|
|
5:GDB is free software, covered by the GNU General Public License, and you are
|
|
6:welcome to change it and/or distribute copies of it under certain conditions.
|
|
7:Type "show copying" to see the conditions.
|
|
8:There is absolutely no warranty for GDB. Type "show warranty" for details.
|
|
9:This GDB was configured as "i386-undermydesk-freebsd"...
|
|
10:panic: page fault
|
|
11:panic messages:
|
|
12:---
|
|
13:Fatal trap 12: page fault while in kernel mode
|
|
14:cpuid = 0; apic id = 00
|
|
15:fault virtual address = 0x300
|
|
16:fault code: = supervisor read, page not present
|
|
17:instruction pointer = 0x8:0xc0713860
|
|
18:stack pointer = 0x10:0xdc1d0b70
|
|
19:frame pointer = 0x10:0xdc1d0b7c
|
|
20:code segment = base 0x0, limit 0xfffff, type 0x1b
|
|
21: = DPL 0, pres 1, def32 1, gran 1
|
|
22:processor eflags = resume, IOPL = 0
|
|
23:current process = 14394 (uname)
|
|
24:trap number = 12
|
|
25:panic: page fault
|
|
26 cpuid = 0;
|
|
27:Stack backtrace:
|
|
28
|
|
29:syncing disks, buffers remaining... 2199 2199 panic: mi_switch: switch in a critical section
|
|
30:cpuid = 0;
|
|
31:Uptime: 2h43m19s
|
|
32:Dumping 255 MB
|
|
33: 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
|
|
34:---
|
|
35:Reading symbols from /boot/kernel/snd_maestro3.ko...done.
|
|
36:Loaded symbols for /boot/kernel/snd_maestro3.ko
|
|
37:Reading symbols from /boot/kernel/snd_pcm.ko...done.
|
|
38:Loaded symbols for /boot/kernel/snd_pcm.ko
|
|
39:#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240
|
|
40:240 dumping++;
|
|
41:<prompt>(kgdb)</prompt> <userinput>list *0xc0713860</userinput>
|
|
42:0xc0713860 is in lapic_ipi_wait (/usr/src/sys/i386/i386/local_apic.c:663).
|
|
43:658 incr = 0;
|
|
44:659 delay = 1;
|
|
45:660 } else
|
|
46:661 incr = 1;
|
|
47:662 for (x = 0; x < delay; x += incr) {
|
|
48:663 if ((lapic->icr_lo & APIC_DELSTAT_MASK) == APIC_DELSTAT_IDLE)
|
|
49:664 return (1);
|
|
50:665 ia32_pause();
|
|
51:666 }
|
|
52:667 return (0);
|
|
53:<prompt>(kgdb)</prompt> <userinput>backtrace</userinput>
|
|
54:#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240
|
|
55:#1 0xc055fd9b in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:372
|
|
56:#2 0xc056019d in panic () at /usr/src/sys/kern/kern_shutdown.c:550
|
|
57:#3 0xc0567ef5 in mi_switch () at /usr/src/sys/kern/kern_synch.c:470
|
|
58:#4 0xc055fa87 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:312
|
|
59:#5 0xc056019d in panic () at /usr/src/sys/kern/kern_shutdown.c:550
|
|
60:#6 0xc0720c66 in trap_fatal (frame=0xdc1d0b30, eva=0)
|
|
61: at /usr/src/sys/i386/i386/trap.c:821
|
|
62:#7 0xc07202b3 in trap (frame=
|
|
63: {tf_fs = -1065484264, tf_es = -1065484272, tf_ds = -1065484272, tf_edi = 1, tf_esi = 0, tf_ebp = -602076292, tf_isp = -602076324, tf_ebx = 0, tf_edx = 0, tf_ecx = 1000000, tf_eax = 243, tf_trapno = 12, tf_err = 0, tf_eip = -1066321824, tf_cs = 8, tf_eflags = 65671, tf_esp = 243, tf_ss = 0})
|
|
64: at /usr/src/sys/i386/i386/trap.c:250
|
|
65:#8 0xc070c9f8 in calltrap () at {standard input}:94
|
|
66:#9 0xc07139f3 in lapic_ipi_vectored (vector=0, dest=0)
|
|
67: at /usr/src/sys/i386/i386/local_apic.c:733
|
|
68:#10 0xc0718b23 in ipi_selected (cpus=1, ipi=1)
|
|
69: at /usr/src/sys/i386/i386/mp_machdep.c:1115
|
|
70:#11 0xc057473e in kseq_notify (ke=0xcc05e360, cpu=0)
|
|
71: at /usr/src/sys/kern/sched_ule.c:520
|
|
72:#12 0xc0575cad in sched_add (td=0xcbcf5c80)
|
|
73: at /usr/src/sys/kern/sched_ule.c:1366
|
|
74:#13 0xc05666c6 in setrunqueue (td=0xcc05e360)
|
|
75: at /usr/src/sys/kern/kern_switch.c:422
|
|
76:#14 0xc05752f4 in sched_wakeup (td=0xcbcf5c80)
|
|
77: at /usr/src/sys/kern/sched_ule.c:999
|
|
78:#15 0xc056816c in setrunnable (td=0xcbcf5c80)
|
|
79: at /usr/src/sys/kern/kern_synch.c:570
|
|
80:#16 0xc0567d53 in wakeup (ident=0xcbcf5c80)
|
|
81: at /usr/src/sys/kern/kern_synch.c:411
|
|
82:#17 0xc05490a8 in exit1 (td=0xcbcf5b40, rv=0)
|
|
83: at /usr/src/sys/kern/kern_exit.c:509
|
|
84:#18 0xc0548011 in sys_exit () at /usr/src/sys/kern/kern_exit.c:102
|
|
85:#19 0xc0720fd0 in syscall (frame=
|
|
86: {tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 0, tf_esi = -1, tf_ebp = -1077940712, tf_isp = -602075788, tf_ebx = 672411944, tf_edx = 10, tf_ecx = 672411600, tf_eax = 1, tf_trapno = 12, tf_err = 2, tf_eip = 671899563, tf_cs = 31, tf_eflags = 642, tf_esp = -1077940740, tf_ss = 47})
|
|
87: at /usr/src/sys/i386/i386/trap.c:1010
|
|
88:#20 0xc070ca4d in Xint0x80_syscall () at {standard input}:136
|
|
89:---Can't read userspace from dump, or kernel process---
|
|
90:<prompt>(kgdb)</prompt> <userinput>quit</userinput></screen></para>
|
|
|
|
|
|
<para>This next trace is an older dump from the FreeBSD 2 time
|
|
frame, but is more involved and demonstrates more of the
|
|
features of <command>gdb</command>. Long lines have been folded
|
|
to improve readability, and the lines are numbered for
|
|
reference. Despite this, it is a real-world error trace taken
|
|
during the development of the pcvt console driver.</para>
|
|
|
|
<screen> 1:Script started on Fri Dec 30 23:15:22 1994
|
|
2:&prompt.root; <userinput>cd /sys/compile/URIAH</userinput>
|
|
3:&prompt.root; <userinput>gdb -k kernel /var/crash/vmcore.1</userinput>
|
|
4:Reading symbol data from /usr/src/sys/compile/URIAH/kernel
|
|
...done.
|
|
5:IdlePTD 1f3000
|
|
6:panic: because you said to!
|
|
7:current pcb at 1e3f70
|
|
8:Reading in symbols for ../../i386/i386/machdep.c...done.
|
|
9:<prompt>(kgdb)</prompt> <userinput>backtrace</userinput>
|
|
10:#0 boot (arghowto=256) (../../i386/i386/machdep.c line 767)
|
|
11:#1 0xf0115159 in panic ()
|
|
12:#2 0xf01955bd in diediedie () (../../i386/i386/machdep.c line 698)
|
|
13:#3 0xf010185e in db_fncall ()
|
|
14:#4 0xf0101586 in db_command (-266509132, -266509516, -267381073)
|
|
15:#5 0xf0101711 in db_command_loop ()
|
|
16:#6 0xf01040a0 in db_trap ()
|
|
17:#7 0xf0192976 in kdb_trap (12, 0, -272630436, -266743723)
|
|
18:#8 0xf019d2eb in trap_fatal (...)
|
|
19:#9 0xf019ce60 in trap_pfault (...)
|
|
20:#10 0xf019cb2f in trap (...)
|
|
21:#11 0xf01932a1 in exception:calltrap ()
|
|
22:#12 0xf0191503 in cnopen (...)
|
|
23:#13 0xf0132c34 in spec_open ()
|
|
24:#14 0xf012d014 in vn_open ()
|
|
25:#15 0xf012a183 in open ()
|
|
26:#16 0xf019d4eb in syscall (...)
|
|
27:<prompt>(kgdb)</prompt> <userinput>up 10</userinput>
|
|
28:Reading in symbols for ../../i386/i386/trap.c...done.
|
|
29:#10 0xf019cb2f in trap (frame={tf_es = -260440048, tf_ds = 16, tf_\
|
|
30:edi = 3072, tf_esi = -266445372, tf_ebp = -272630356, tf_isp = -27\
|
|
31:2630396, tf_ebx = -266427884, tf_edx = 12, tf_ecx = -266427884, tf\
|
|
32:_eax = 64772224, tf_trapno = 12, tf_err = -272695296, tf_eip = -26\
|
|
33:6672343, tf_cs = -266469368, tf_eflags = 66066, tf_esp = 3072, tf_\
|
|
34:ss = -266427884}) (../../i386/i386/trap.c line 283)
|
|
35:283 (void) trap_pfault(&frame, FALSE);
|
|
36:<prompt>(kgdb)</prompt> <userinput>frame frame->tf_ebp frame->tf_eip</userinput>
|
|
37:Reading in symbols for ../../i386/isa/pcvt/pcvt_drv.c...done.
|
|
38:#0 0xf01ae729 in pcopen (dev=3072, flag=3, mode=8192, p=(struct p\
|
|
39:roc *) 0xf07c0c00) (../../i386/isa/pcvt/pcvt_drv.c line 403)
|
|
40:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
|
|
41:<prompt>(kgdb)</prompt> <userinput>list</userinput>
|
|
42:398
|
|
43:399 tp->t_state |= TS_CARR_ON;
|
|
44:400 tp->t_cflag |= CLOCAL; /* cannot be a modem (:-) */
|
|
45:401
|
|
46:402 #if PCVT_NETBSD || (PCVT_FREEBSD >= 200)
|
|
47:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
|
|
48:404 #else
|
|
49:405 return ((*linesw[tp->t_line].l_open)(dev, tp, flag));
|
|
50:406 #endif /* PCVT_NETBSD || (PCVT_FREEBSD >= 200) */
|
|
51:407 }
|
|
52:<prompt>(kgdb)</prompt> <userinput>print tp</userinput>
|
|
53:Reading in symbols for ../../i386/i386/cons.c...done.
|
|
54:$1 = (struct tty *) 0x1bae
|
|
55:<prompt>(kgdb)</prompt> <userinput>print tp->t_line</userinput>
|
|
56:$2 = 1767990816
|
|
57:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
58:#1 0xf0191503 in cnopen (dev=0x00000000, flag=3, mode=8192, p=(st\
|
|
59:ruct proc *) 0xf07c0c00) (../../i386/i386/cons.c line 126)
|
|
60: return ((*cdevsw[major(dev)].d_open)(dev, flag, mode, p));
|
|
61:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
62:#2 0xf0132c34 in spec_open ()
|
|
63:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
64:#3 0xf012d014 in vn_open ()
|
|
65:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
66:#4 0xf012a183 in open ()
|
|
67:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
68:#5 0xf019d4eb in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi =\
|
|
69: 2158592, tf_esi = 0, tf_ebp = -272638436, tf_isp = -272629788, tf\
|
|
70:_ebx = 7086, tf_edx = 1, tf_ecx = 0, tf_eax = 5, tf_trapno = 582, \
|
|
71:tf_err = 582, tf_eip = 75749, tf_cs = 31, tf_eflags = 582, tf_esp \
|
|
72:= -272638456, tf_ss = 39}) (../../i386/i386/trap.c line 673)
|
|
73:673 error = (*callp->sy_call)(p, args, rval);
|
|
74:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
75:Initial frame selected; you cannot go up.
|
|
76:<prompt>(kgdb)</prompt> <userinput>quit</userinput></screen>
|
|
<para>Comments to the above script:</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>line 6:</term>
|
|
|
|
<listitem>
|
|
<para>This is a dump taken from within DDB (see below), hence the
|
|
panic comment <quote>because you said to!</quote>, and a rather
|
|
long stack trace; the initial reason for going into DDB has been a
|
|
page fault trap though.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>line 20:</term>
|
|
|
|
<listitem>
|
|
<para>This is the location of function <function>trap()</function>
|
|
in the stack trace.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>line 36:</term>
|
|
|
|
<listitem>
|
|
<para>Force usage of a new stack frame; this is no longer necessary.
|
|
The stack frames are supposed to point to the right
|
|
locations now, even in case of a trap.
|
|
From looking at the code in source line 403, there is a
|
|
high probability that either the pointer access for
|
|
<quote>tp</quote> was messed up, or the array access was out of
|
|
bounds.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>line 52:</term>
|
|
|
|
<listitem>
|
|
<para>The pointer looks suspicious, but happens to be a valid
|
|
address.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>line 56:</term>
|
|
|
|
<listitem>
|
|
<para>However, it obviously points to garbage, so we have found our
|
|
error! (For those unfamiliar with that particular piece of code:
|
|
<literal>tp->t_line</literal> refers to the line discipline of
|
|
the console device here, which must be a rather small integer
|
|
number.)</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<tip><para>If your system is crashing regularly and you're running
|
|
out of disk space, deleting old <filename>vmcore</filename>
|
|
files in <filename>/var/crash</filename> could save a
|
|
considerable amount of disk space!</para></tip>
|
|
</sect1>
|
|
|
|
<sect1 id="kerneldebug-ddd">
|
|
<title>Debugging a Crash Dump with DDD</title>
|
|
|
|
<para>Examining a kernel crash dump with a graphical debugger like
|
|
<command>ddd</command> is also possible (you will need to install
|
|
the <filename role="package">devel/ddd</filename> port in order to use the
|
|
<command>ddd</command> debugger). Add the <option>-k</option>
|
|
option to the <command>ddd</command> command line you would use
|
|
normally. For example;</para>
|
|
|
|
<screen>&prompt.root; <userinput>ddd -k /var/crash/kernel.0 /var/crash/vmcore.0</userinput></screen>
|
|
|
|
<para>You should then be able to go about looking at the crash dump using
|
|
<command>ddd</command>'s graphical interface.</para>
|
|
</sect1>
|
|
|
|
<sect1 id="kerneldebug-post-mortem">
|
|
<title>Post-Mortem Analysis of a Dump</title>
|
|
|
|
<para>What do you do if a kernel dumped core but you did not expect it,
|
|
and it is therefore not compiled using <command>config -g</command>? Not
|
|
everything is lost here. Do not panic!</para>
|
|
|
|
<para>Of course, you still need to enable crash dumps. See above for the
|
|
options you have to specify in order to do this.</para>
|
|
|
|
<para>Go to your kernel config directory
|
|
(<filename>/usr/src/sys/<replaceable>arch</replaceable>/conf</filename>)
|
|
and edit your configuration file. Uncomment (or add, if it does not
|
|
exist) the following line:</para>
|
|
|
|
<programlisting>makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols</programlisting>
|
|
|
|
<para>Rebuild the kernel. Due to the time stamp change on the Makefile,
|
|
some other object files will be rebuilt, for example
|
|
<filename>trap.o</filename>. With a bit of luck, the added
|
|
<option>-g</option> option will not change anything for the generated
|
|
code, so you will finally get a new kernel with similar code to the
|
|
faulting one but some debugging symbols. You should at least verify the
|
|
old and new sizes with the &man.size.1; command. If there is a
|
|
mismatch, you probably need to give up here.</para>
|
|
|
|
<para>Go and examine the dump as described above. The debugging symbols
|
|
might be incomplete for some places, as can be seen in the stack trace
|
|
in the example above where some functions are displayed without line
|
|
numbers and argument lists. If you need more debugging symbols, remove
|
|
the appropriate object files, recompile the kernel again and repeat the
|
|
<command>gdb <option>-k</option></command>
|
|
session until you know enough.</para>
|
|
|
|
<para>All this is not guaranteed to work, but it will do it fine in most
|
|
cases.</para>
|
|
</sect1>
|
|
|
|
<sect1 id="kerneldebug-online-ddb">
|
|
<title>On-Line Kernel Debugging Using DDB</title>
|
|
|
|
<para>While <command>gdb <option>-k</option></command> as an off-line debugger provides a very
|
|
high level of user interface, there are some things it cannot do. The
|
|
most important ones being breakpointing and single-stepping kernel
|
|
code.</para>
|
|
|
|
<para>If you need to do low-level debugging on your kernel, there is an
|
|
on-line debugger available called DDB. It allows setting of
|
|
breakpoints, single-stepping kernel functions, examining and changing
|
|
kernel variables, etc. However, it cannot access kernel source files,
|
|
and only has access to the global and static symbols, not to the full
|
|
debug information like <command>gdb</command> does.</para>
|
|
|
|
<para>To configure your kernel to include DDB, add the option line
|
|
|
|
<programlisting>options DDB</programlisting>
|
|
|
|
to your config file, and rebuild. (See <ulink
|
|
url="&url.books.handbook;/index.html">The FreeBSD Handbook</ulink> for details on
|
|
configuring the FreeBSD kernel).</para>
|
|
|
|
<note>
|
|
<para>If you have an older version of the boot blocks, your
|
|
debugger symbols might not be loaded at all. Update the boot blocks;
|
|
the recent ones load the DDB symbols automatically.</para>
|
|
</note>
|
|
|
|
<para>Once your DDB kernel is running, there are several ways to enter
|
|
DDB. The first, and earliest way is to type the boot flag
|
|
<option>-d</option> right at the boot prompt. The kernel will start up
|
|
in debug mode and enter DDB prior to any device probing. Hence you can
|
|
even debug the device probe/attach functions.</para>
|
|
|
|
<para>The second scenario is to drop to the debugger once the
|
|
system has booted. There are two simple ways to accomplish
|
|
this. If you would like to break to the debugger from the
|
|
command prompt, simply type the command:</para>
|
|
|
|
<screen>&prompt.root; <userinput>sysctl debug.enter_debugger=ddb</userinput></screen>
|
|
|
|
<para>Alternatively, if you are at the system console, you may use
|
|
a hot-key on the keyboard. The default break-to-debugger
|
|
sequence is <keycombo action="simul"><keycap>Ctrl</keycap>
|
|
<keycap>Alt</keycap><keycap>ESC</keycap></keycombo>. For
|
|
syscons, this sequence can be remapped and some of the
|
|
distributed maps out there do this, so check to make sure you
|
|
know the right sequence to use. There is an option available
|
|
for serial consoles that allows the use of a serial line BREAK on the
|
|
console line to enter DDB (<literal>options BREAK_TO_DEBUGGER</literal>
|
|
in the kernel config file). It is not the default since there are a lot
|
|
of serial adapters around that gratuitously generate a BREAK
|
|
condition, for example when pulling the cable.</para>
|
|
|
|
<para>The third way is that any panic condition will branch to DDB if the
|
|
kernel is configured to use it. For this reason, it is not wise to
|
|
configure a kernel with DDB for a machine running unattended.</para>
|
|
|
|
<para>The DDB commands roughly resemble some <command>gdb</command>
|
|
commands. The first thing you probably need to do is to set a
|
|
breakpoint:</para>
|
|
|
|
<screen><userinput>b function-name</userinput>
|
|
<userinput>b address</userinput></screen>
|
|
|
|
<para>Numbers are taken hexadecimal by default, but to make them distinct
|
|
from symbol names; hexadecimal numbers starting with the letters
|
|
<literal>a-f</literal> need to be preceded with <literal>0x</literal>
|
|
(this is optional for other numbers). Simple expressions are allowed,
|
|
for example: <literal>function-name + 0x103</literal>.</para>
|
|
|
|
<para>To continue the operation of an interrupted kernel, simply
|
|
type:</para>
|
|
|
|
<screen><userinput>c</userinput></screen>
|
|
|
|
<para>To get a stack trace, use:</para>
|
|
|
|
<screen><userinput>trace</userinput></screen>
|
|
|
|
<note>
|
|
<para>Note that when entering DDB via a hot-key, the kernel is currently
|
|
servicing an interrupt, so the stack trace might be not of much use
|
|
to you.</para>
|
|
</note>
|
|
|
|
<para>If you want to remove a breakpoint, use</para>
|
|
|
|
|
|
<screen><userinput>del</userinput>
|
|
<userinput>del address-expression</userinput></screen>
|
|
|
|
<para>The first form will be accepted immediately after a breakpoint hit,
|
|
and deletes the current breakpoint. The second form can remove any
|
|
breakpoint, but you need to specify the exact address; this can be
|
|
obtained from:</para>
|
|
|
|
<screen><userinput>show b</userinput></screen>
|
|
|
|
<para>To single-step the kernel, try:</para>
|
|
|
|
<screen><userinput>s</userinput></screen>
|
|
|
|
<para>This will step into functions, but you can make DDB trace them until
|
|
the matching return statement is reached by:</para>
|
|
|
|
<screen><userinput>n</userinput></screen>
|
|
|
|
<note>
|
|
<para>This is different from <command>gdb</command>'s
|
|
<command>next</command> statement; it is like <command>gdb</command>'s
|
|
<command>finish</command>.</para>
|
|
</note>
|
|
|
|
<para>To examine data from memory, use (for example):
|
|
|
|
<screen><userinput>x/wx 0xf0133fe0,40</userinput>
|
|
<userinput>x/hd db_symtab_space</userinput>
|
|
<userinput>x/bc termbuf,10</userinput>
|
|
<userinput>x/s stringbuf</userinput></screen>
|
|
|
|
for word/halfword/byte access, and hexadecimal/decimal/character/ string
|
|
display. The number after the comma is the object count. To display
|
|
the next 0x10 items, simply use:</para>
|
|
|
|
<screen><userinput>x ,10</userinput></screen>
|
|
|
|
<para>Similarly, use
|
|
|
|
<screen><userinput>x/ia foofunc,10</userinput></screen>
|
|
|
|
to disassemble the first 0x10 instructions of
|
|
<function>foofunc</function>, and display them along with their offset
|
|
from the beginning of <function>foofunc</function>.</para>
|
|
|
|
<para>To modify memory, use the write command:</para>
|
|
|
|
<screen><userinput>w/b termbuf 0xa 0xb 0</userinput>
|
|
<userinput>w/w 0xf0010030 0 0</userinput></screen>
|
|
|
|
<para>The command modifier
|
|
(<literal>b</literal>/<literal>h</literal>/<literal>w</literal>)
|
|
specifies the size of the data to be written, the first following
|
|
expression is the address to write to and the remainder is interpreted
|
|
as data to write to successive memory locations.</para>
|
|
|
|
<para>If you need to know the current registers, use:</para>
|
|
|
|
<screen><userinput>show reg</userinput></screen>
|
|
|
|
<para>Alternatively, you can display a single register value by e.g.
|
|
|
|
<screen><userinput>p $eax</userinput></screen>
|
|
|
|
and modify it by:</para>
|
|
|
|
<screen><userinput>set $eax new-value</userinput></screen>
|
|
|
|
<para>Should you need to call some kernel functions from DDB, simply
|
|
say:</para>
|
|
|
|
<screen><userinput>call func(arg1, arg2, ...)</userinput></screen>
|
|
|
|
<para>The return value will be printed.</para>
|
|
|
|
<para>For a &man.ps.1; style summary of all running processes, use:</para>
|
|
|
|
<screen><userinput>ps</userinput></screen>
|
|
|
|
<para>Now you have examined why your kernel failed, and you wish to
|
|
reboot. Remember that, depending on the severity of previous
|
|
malfunctioning, not all parts of the kernel might still be working as
|
|
expected. Perform one of the following actions to shut down and reboot
|
|
your system:</para>
|
|
|
|
<screen><userinput>panic</userinput></screen>
|
|
|
|
<para>This will cause your kernel to dump core and reboot, so you can
|
|
later analyze the core on a higher level with <command>gdb</command>. This command
|
|
usually must be followed by another <command>continue</command>
|
|
statement.</para>
|
|
|
|
<screen><userinput>call boot(0)</userinput></screen>
|
|
|
|
<para>Which might be a good way to cleanly shut down the running system,
|
|
<function>sync()</function> all disks, and finally reboot. As long as
|
|
the disk and filesystem interfaces of the kernel are not damaged, this
|
|
might be a good way for an almost clean shutdown.</para>
|
|
|
|
<screen><userinput>call cpu_reset()</userinput></screen>
|
|
|
|
<para>This is the final way out of disaster and almost the same as hitting the
|
|
Big Red Button.</para>
|
|
|
|
<para>If you need a short command summary, simply type:</para>
|
|
|
|
<screen><userinput>help</userinput></screen>
|
|
|
|
<para>However, it is highly recommended to have a printed copy of the
|
|
&man.ddb.4; manual page ready for a debugging
|
|
session. Remember that it is hard to read the on-line manual while
|
|
single-stepping the kernel.</para>
|
|
</sect1>
|
|
|
|
<sect1 id="kerneldebug-online-gdb">
|
|
<title>On-Line Kernel Debugging Using Remote GDB</title>
|
|
|
|
<para>This feature has been supported since FreeBSD 2.2, and it is
|
|
actually a very neat one.</para>
|
|
|
|
<para>GDB has already supported <emphasis>remote debugging</emphasis> for
|
|
a long time. This is done using a very simple protocol along a serial
|
|
line. Unlike the other methods described above, you will need two
|
|
machines for doing this. One is the host providing the debugging
|
|
environment, including all the sources, and a copy of the kernel binary
|
|
with all the symbols in it, and the other one is the target machine that
|
|
simply runs a similar copy of the very same kernel (but stripped of the
|
|
debugging information).</para>
|
|
|
|
<para>You should configure the kernel in question with <command>config
|
|
-g</command>, include <option>DDB</option> into the configuration, and
|
|
compile it as usual. This gives a large binary, due to the
|
|
debugging information. Copy this kernel to the target machine, strip
|
|
the debugging symbols off with <command>strip -x</command>, and boot it
|
|
using the <option>-d</option> boot option. Connect the serial line
|
|
of the target machine that has "flags 080" set on its sio device
|
|
to any serial line of the debugging host.
|
|
Now, on the debugging machine, go to the compile directory of the target
|
|
kernel, and start <command>gdb</command>:</para>
|
|
|
|
<screen>&prompt.user; <userinput>gdb -k kernel</userinput>
|
|
GDB is free software and you are welcome to distribute copies of it
|
|
under certain conditions; type "show copying" to see the conditions.
|
|
There is absolutely no warranty for GDB; type "show warranty" for details.
|
|
GDB 4.16 (i386-unknown-freebsd),
|
|
Copyright 1996 Free Software Foundation, Inc...
|
|
<prompt>(kgdb)</prompt> </screen>
|
|
|
|
<para>Initialize the remote debugging session (assuming the first serial
|
|
port is being used) by:</para>
|
|
|
|
<screen><prompt>(kgdb)</prompt> <userinput>target remote /dev/cuaa0</userinput></screen>
|
|
|
|
<para>Now, on the target host (the one that entered DDB right before even
|
|
starting the device probe), type:</para>
|
|
|
|
<screen>Debugger("Boot flags requested debugger")
|
|
Stopped at Debugger+0x35: movb $0, edata+0x51bc
|
|
<prompt>db></prompt> <userinput>gdb</userinput></screen>
|
|
|
|
<para>DDB will respond with:</para>
|
|
|
|
<screen>Next trap will enter GDB remote protocol mode</screen>
|
|
|
|
<para>Every time you type <command>gdb</command>, the mode will be toggled
|
|
between remote GDB and local DDB. In order to force a next trap
|
|
immediately, simply type <command>s</command> (step). Your hosting GDB
|
|
will now gain control over the target kernel:</para>
|
|
|
|
<screen>Remote debugging using /dev/cuaa0
|
|
Debugger (msg=0xf01b0383 "Boot flags requested debugger")
|
|
at ../../i386/i386/db_interface.c:257
|
|
<prompt>(kgdb)</prompt></screen>
|
|
|
|
<para>You can use this session almost as any other GDB session, including
|
|
full access to the source, running it in gud-mode inside an Emacs window
|
|
(which gives you an automatic source code display in another Emacs
|
|
window), etc.</para>
|
|
</sect1>
|
|
|
|
<sect1 id="kerneldebug-kld">
|
|
<title>Debugging Loadable Modules Using GDB</title>
|
|
|
|
<para>When debugging a panic that occurred within a module, or
|
|
using remote GDB against a machine that uses dynamic modules,
|
|
you need to tell GDB how to obtain symbol information for those
|
|
modules.</para>
|
|
|
|
<para>First, you need to build the module(s) with debugging
|
|
information:</para>
|
|
|
|
<screen>&prompt.root; <userinput>cd /sys/modules/linux</userinput>
|
|
&prompt.root; <userinput>make clean; make COPTS=-g</userinput></screen>
|
|
|
|
<para>If you are using remote GDB, you can run
|
|
<command>kldstat</command> on the target machine to find out
|
|
where the module was loaded:</para>
|
|
|
|
<screen>&prompt.root; <userinput>kldstat</userinput>
|
|
Id Refs Address Size Name
|
|
1 4 0xc0100000 1c1678 kernel
|
|
2 1 0xc0a9e000 6000 linprocfs.ko
|
|
3 1 0xc0ad7000 2000 warp_saver.ko
|
|
4 1 0xc0adc000 11000 linux.ko</screen>
|
|
|
|
<para>If you are debugging a crash dump, you will need to walk the
|
|
<literal>linker_files</literal> list, starting at
|
|
<literal>linker_files->tqh_first</literal> and following the
|
|
<literal>link.tqe_next</literal> pointers until you find the
|
|
entry with the <literal>filename</literal> you are looking for.
|
|
The <literal>address</literal> member of that entry is the load
|
|
address of the module.</para>
|
|
|
|
<para>Next, you need to find out the offset of the text section
|
|
within the module:</para>
|
|
|
|
<screen>&prompt.root; <userinput>objdump --section-headers /sys/modules/linux/linux.ko | grep text</userinput>
|
|
3 .rel.text 000016e0 000038e0 000038e0 000038e0 2**2
|
|
10 .text 00007f34 000062d0 000062d0 000062d0 2**2</screen>
|
|
|
|
<para>The one you want is the <literal>.text</literal> section,
|
|
section 10 in the above example. The fourth hexadecimal field
|
|
(sixth field overall) is the offset of the text section within
|
|
the file. Add this offset to the load address of the module to
|
|
obtain the relocation address for the module's code. In our
|
|
example, we get 0xc0adc000 + 0x62d0 = 0xc0ae22d0. Use the
|
|
<command>add-symbol-file</command> command in GDB to tell the
|
|
debugger about the module:</para>
|
|
|
|
<screen><prompt>(kgdb)</prompt> <userinput>add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0</userinput>
|
|
add symbol table from file "/sys/modules/linux/linux.ko" at text_addr = 0xc0ae22d0?
|
|
(y or n) <userinput>y</userinput>
|
|
Reading symbols from /sys/modules/linux/linux.ko...done.
|
|
<prompt>(kgdb)</prompt></screen>
|
|
|
|
<para>You should now have access to all the symbols in the
|
|
module.</para>
|
|
</sect1>
|
|
|
|
<sect1 id="kerneldebug-console">
|
|
<title>Debugging a Console Driver</title>
|
|
|
|
<para>Since you need a console driver to run DDB on, things are more
|
|
complicated if the console driver itself is failing. You might remember
|
|
the use of a serial console (either with modified boot blocks, or by
|
|
specifying <option>-h</option> at the <prompt>Boot:</prompt> prompt),
|
|
and hook up a standard terminal onto your first serial port. DDB works
|
|
on any configured console driver, including a serial
|
|
console.</para>
|
|
</sect1>
|
|
</chapter>
|
|
|
|
<!--
|
|
Local Variables:
|
|
mode: sgml
|
|
sgml-declaration: "../chapter.decl"
|
|
sgml-indent-data: t
|
|
sgml-omittag: nil
|
|
sgml-always-quote-attributes: t
|
|
sgml-parent-document: ("../book.sgml" "part" "chapter")
|
|
End:
|
|
-->
|
|
|