* Minor markup fix. * Couple of grammatical/readability improvements. * Mention that ddd is a port. PR: docs/34921 Submitted by: Ceri <setantae@submonkey.net>
649 lines
28 KiB
Text
649 lines
28 KiB
Text
<!--
|
|
The FreeBSD Documentation Project
|
|
|
|
$FreeBSD$
|
|
-->
|
|
|
|
<chapter id="kerneldebug">
|
|
<title>Kernel Debugging</title>
|
|
|
|
<para><emphasis>Contributed by &a.paul; and &a.joerg;</emphasis></para>
|
|
|
|
<sect1>
|
|
<title>Debugging a Kernel Crash Dump with <command>gdb</command></title>
|
|
|
|
<para>Here are some instructions for getting kernel debugging
|
|
working on a crash dump. They assume that you have enough swap
|
|
space for a crash dump. Typically you want to
|
|
specify one of the swap devices specified in
|
|
<filename>/etc/fstab</filename>. Dumps to non-swap devices,
|
|
tapes for example, are currently not supported.</para>
|
|
|
|
<note>
|
|
<para>Use the &man.dumpon.8; command to tell the kernel where to dump to
|
|
(note that this will have to be done after configuring the partition in
|
|
question as swap space via &man.swapon.8;). This is normally arranged
|
|
by setting the <varname>dumpdev</varname> variable in
|
|
<filename>/etc/rc.conf</filename>.</para>
|
|
|
|
<para>Alternatively, you can hard-code the dump device via the
|
|
<literal>dump</literal> clause in the <literal>config</literal> line of
|
|
your kernel configuration file. This approach is deprecated and should
|
|
be used only if you want a crash dump from a kernel that crashes during
|
|
booting.</para>
|
|
</note>
|
|
|
|
<note>
|
|
<para>In the following, the term <command>gdb</command> refers to
|
|
the debugger <command>gdb</command> run in <quote>kernel debug
|
|
mode</quote>. This can be accomplished by starting the
|
|
<command>gdb</command> with the option <option>-k</option>. In
|
|
kernel debug mode, <command>gdb</command> changes its prompt to
|
|
<prompt>(kgdb)</prompt>.</para>
|
|
</note>
|
|
|
|
<tip>
|
|
<para>If you are using FreeBSD 3 or earlier, you should make a stripped
|
|
copy of the debug kernel, rather than installing the large debug
|
|
kernel itself:</para>
|
|
|
|
<screen>&prompt.root; <userinput>cp kernel kernel.debug</userinput>
|
|
&prompt.root; <userinput>strip -g kernel</userinput></screen>
|
|
|
|
<para>This stage is not necessary, but it is recommended. (In
|
|
FreeBSD 4 and later releases this step is performed automatically
|
|
at the end of the kernel <command>make</command> process.)
|
|
When the kernel has been stripped, either automatically or by
|
|
using the commands above, you may install it as usual by typing
|
|
<command>make install</command>.</para>
|
|
|
|
<para>Note that older releases of FreeBSD (up to but not including
|
|
3.1) used a.out kernels by default, which must have their symbol
|
|
tables permanently resident in physical memory. With the larger
|
|
symbol table in an unstripped debug kernel, this is wasteful.
|
|
Recent FreeBSD releases use ELF kernels where this is no longer a
|
|
problem.</para>
|
|
</tip>
|
|
|
|
<para>If you are testing a new kernel, for example by typing the new
|
|
kernel's name at the boot prompt, but need to boot a different one in
|
|
order to get your system up and running again, boot it only into single
|
|
user state using the <option>-s</option> flag at the boot prompt, and
|
|
then perform the following steps:</para>
|
|
|
|
<screen>&prompt.root; <userinput>fsck -p</userinput>
|
|
&prompt.root; <userinput>mount -a -t ufs</userinput> # so your file system for /var/crash is writable
|
|
&prompt.root; <userinput>savecore -N /kernel.panicked /var/crash</userinput>
|
|
&prompt.root; <userinput>exit</userinput> # ...to multi-user</screen>
|
|
|
|
<para>This instructs &man.savecore.8; to use another kernel for symbol
|
|
name extraction. It would otherwise default to the currently running
|
|
kernel and most likely not do anything at all since the crash dump and
|
|
the kernel symbols differ.</para>
|
|
|
|
<para>Now, after a crash dump, go to
|
|
<filename>/sys/compile/WHATEVER</filename> and run
|
|
<command>gdb <option>-k</option></command>. From <command>gdb</command> do:
|
|
|
|
<screen><userinput>symbol-file kernel.debug</userinput>
|
|
<userinput>exec-file /var/crash/kernel.0</userinput>
|
|
<userinput>core-file /var/crash/vmcore.0</userinput></screen>
|
|
|
|
and voila, you can debug the crash dump using the kernel sources just
|
|
like you can for any other program.</para>
|
|
|
|
<para>Here is a script log of a <command>gdb</command> session
|
|
illustrating the procedure. Long lines have been folded to improve
|
|
readability, and the lines are numbered for reference. Despite this, it
|
|
is a real-world error trace taken during the development of the pcvt
|
|
console driver.</para>
|
|
|
|
<screen> 1:Script started on Fri Dec 30 23:15:22 1994
|
|
2:&prompt.root; <userinput>cd /sys/compile/URIAH</userinput>
|
|
3:&prompt.root; <userinput>gdb -k kernel /var/crash/vmcore.1</userinput>
|
|
4:Reading symbol data from /usr/src/sys/compile/URIAH/kernel
|
|
...done.
|
|
5:IdlePTD 1f3000
|
|
6:panic: because you said to!
|
|
7:current pcb at 1e3f70
|
|
8:Reading in symbols for ../../i386/i386/machdep.c...done.
|
|
9:<prompt>(kgdb)</prompt> <userinput>where</userinput>
|
|
10:#0 boot (arghowto=256) (../../i386/i386/machdep.c line 767)
|
|
11:#1 0xf0115159 in panic ()
|
|
12:#2 0xf01955bd in diediedie () (../../i386/i386/machdep.c line 698)
|
|
13:#3 0xf010185e in db_fncall ()
|
|
14:#4 0xf0101586 in db_command (-266509132, -266509516, -267381073)
|
|
15:#5 0xf0101711 in db_command_loop ()
|
|
16:#6 0xf01040a0 in db_trap ()
|
|
17:#7 0xf0192976 in kdb_trap (12, 0, -272630436, -266743723)
|
|
18:#8 0xf019d2eb in trap_fatal (...)
|
|
19:#9 0xf019ce60 in trap_pfault (...)
|
|
20:#10 0xf019cb2f in trap (...)
|
|
21:#11 0xf01932a1 in exception:calltrap ()
|
|
22:#12 0xf0191503 in cnopen (...)
|
|
23:#13 0xf0132c34 in spec_open ()
|
|
24:#14 0xf012d014 in vn_open ()
|
|
25:#15 0xf012a183 in open ()
|
|
26:#16 0xf019d4eb in syscall (...)
|
|
27:<prompt>(kgdb)</prompt> <userinput>up 10</userinput>
|
|
28:Reading in symbols for ../../i386/i386/trap.c...done.
|
|
29:#10 0xf019cb2f in trap (frame={tf_es = -260440048, tf_ds = 16, tf_\
|
|
30:edi = 3072, tf_esi = -266445372, tf_ebp = -272630356, tf_isp = -27\
|
|
31:2630396, tf_ebx = -266427884, tf_edx = 12, tf_ecx = -266427884, tf\
|
|
32:_eax = 64772224, tf_trapno = 12, tf_err = -272695296, tf_eip = -26\
|
|
33:6672343, tf_cs = -266469368, tf_eflags = 66066, tf_esp = 3072, tf_\
|
|
34:ss = -266427884}) (../../i386/i386/trap.c line 283)
|
|
35:283 (void) trap_pfault(&frame, FALSE);
|
|
36:<prompt>(kgdb)</prompt> <userinput>frame frame->tf_ebp frame->tf_eip</userinput>
|
|
37:Reading in symbols for ../../i386/isa/pcvt/pcvt_drv.c...done.
|
|
38:#0 0xf01ae729 in pcopen (dev=3072, flag=3, mode=8192, p=(struct p\
|
|
39:roc *) 0xf07c0c00) (../../i386/isa/pcvt/pcvt_drv.c line 403)
|
|
40:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
|
|
41:<prompt>(kgdb)</prompt> <userinput>list</userinput>
|
|
42:398
|
|
43:399 tp->t_state |= TS_CARR_ON;
|
|
44:400 tp->t_cflag |= CLOCAL; /* cannot be a modem (:-) */
|
|
45:401
|
|
46:402 #if PCVT_NETBSD || (PCVT_FREEBSD >= 200)
|
|
47:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
|
|
48:404 #else
|
|
49:405 return ((*linesw[tp->t_line].l_open)(dev, tp, flag));
|
|
50:406 #endif /* PCVT_NETBSD || (PCVT_FREEBSD >= 200) */
|
|
51:407 }
|
|
52:<prompt>(kgdb)</prompt> <userinput>print tp</userinput>
|
|
53:Reading in symbols for ../../i386/i386/cons.c...done.
|
|
54:$1 = (struct tty *) 0x1bae
|
|
55:<prompt>(kgdb)</prompt> <userinput>print tp->t_line</userinput>
|
|
56:$2 = 1767990816
|
|
57:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
58:#1 0xf0191503 in cnopen (dev=0x00000000, flag=3, mode=8192, p=(st\
|
|
59:ruct proc *) 0xf07c0c00) (../../i386/i386/cons.c line 126)
|
|
60: return ((*cdevsw[major(dev)].d_open)(dev, flag, mode, p));
|
|
61:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
62:#2 0xf0132c34 in spec_open ()
|
|
63:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
64:#3 0xf012d014 in vn_open ()
|
|
65:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
66:#4 0xf012a183 in open ()
|
|
67:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
68:#5 0xf019d4eb in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi =\
|
|
69: 2158592, tf_esi = 0, tf_ebp = -272638436, tf_isp = -272629788, tf\
|
|
70:_ebx = 7086, tf_edx = 1, tf_ecx = 0, tf_eax = 5, tf_trapno = 582, \
|
|
71:tf_err = 582, tf_eip = 75749, tf_cs = 31, tf_eflags = 582, tf_esp \
|
|
72:= -272638456, tf_ss = 39}) (../../i386/i386/trap.c line 673)
|
|
73:673 error = (*callp->sy_call)(p, args, rval);
|
|
74:<prompt>(kgdb)</prompt> <userinput>up</userinput>
|
|
75:Initial frame selected; you cannot go up.
|
|
76:<prompt>(kgdb)</prompt> <userinput>quit</userinput>
|
|
77:&prompt.root; <userinput>exit</userinput>
|
|
78:exit
|
|
79:
|
|
80:Script done on Fri Dec 30 23:18:04 1994</screen>
|
|
<para>Comments to the above script:</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>line 6:</term>
|
|
|
|
<listitem>
|
|
<para>This is a dump taken from within DDB (see below), hence the
|
|
panic comment <quote>because you said to!</quote>, and a rather
|
|
long stack trace; the initial reason for going into DDB has been a
|
|
page fault trap though.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>line 20:</term>
|
|
|
|
<listitem>
|
|
<para>This is the location of function <function>trap()</function>
|
|
in the stack trace.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>line 36:</term>
|
|
|
|
<listitem>
|
|
<para>Force usage of a new stack frame; this is no longer necessary.
|
|
The stack frames are supposed to point to the right
|
|
locations now, even in case of a trap.
|
|
From looking at the code in source line 403, there is a
|
|
high probability that either the pointer access for
|
|
<quote>tp</quote> was messed up, or the array access was out of
|
|
bounds.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>line 52:</term>
|
|
|
|
<listitem>
|
|
<para>The pointer looks suspicious, but happens to be a valid
|
|
address.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>line 56:</term>
|
|
|
|
<listitem>
|
|
<para>However, it obviously points to garbage, so we have found our
|
|
error! (For those unfamiliar with that particular piece of code:
|
|
<literal>tp->t_line</literal> refers to the line discipline of
|
|
the console device here, which must be a rather small integer
|
|
number.)</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>Debugging a Crash Dump with DDD</title>
|
|
|
|
<para>Examining a kernel crash dump with a graphical debugger like
|
|
<command>ddd</command> is also possible (you will need to install
|
|
the <port>devel/ddd</port> port in order to use the
|
|
<command>ddd</command> debugger). Add the <option>-k</option>
|
|
option to the <command>ddd</command> command line you would use
|
|
normally. For example;</para>
|
|
|
|
<screen>&prompt.root; <userinput>ddd -k /var/crash/kernel.0 /var/crash/vmcore.0</userinput></screen>
|
|
|
|
<para>You should then be able to go about looking at the crash dump using
|
|
<command>ddd</command>'s graphical interface.</para>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>Post-Mortem Analysis of a Dump</title>
|
|
|
|
<para>What do you do if a kernel dumped core but you did not expect it,
|
|
and it is therefore not compiled using <command>config -g</command>? Not
|
|
everything is lost here. Do not panic!</para>
|
|
|
|
<para>Of course, you still need to enable crash dumps. See above for the
|
|
options you have to specify in order to do this.</para>
|
|
|
|
<para>Go to your kernel config directory
|
|
(<filename>/usr/src/sys/<replaceable>arch</replaceable>/conf</filename>)
|
|
and edit your configuration file. Uncomment (or add, if it does not
|
|
exist) the following line:</para>
|
|
|
|
<programlisting>makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols</programlisting>
|
|
|
|
<para>Rebuild the kernel. Due to the time stamp change on the Makefile,
|
|
there some other object files will be rebuilt, for example
|
|
<filename>trap.o</filename>. With a bit of luck, the added
|
|
<option>-g</option> option will not change anything for the generated
|
|
code, so you will finally get a new kernel with similar code to the
|
|
faulting one but some debugging symbols. You should at least verify the
|
|
old and new sizes with the &man.size.1; command. If there is a
|
|
mismatch, you probably need to give up here.</para>
|
|
|
|
<para>Go and examine the dump as described above. The debugging symbols
|
|
might be incomplete for some places, as can be seen in the stack trace
|
|
in the example above where some functions are displayed without line
|
|
numbers and argument lists. If you need more debugging symbols, remove
|
|
the appropriate object files, recompile the kernel again and repeat the
|
|
<command>gdb <option>-k</option></command>
|
|
session until you know enough.</para>
|
|
|
|
<para>All this is not guaranteed to work, but it will do it fine in most
|
|
cases.</para>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>On-Line Kernel Debugging Using DDB</title>
|
|
|
|
<para>While <command>gdb <option>-k</option></command> as an off-line debugger provides a very
|
|
high level of user interface, there are some things it cannot do. The
|
|
most important ones being breakpointing and single-stepping kernel
|
|
code.</para>
|
|
|
|
<para>If you need to do low-level debugging on your kernel, there is an
|
|
on-line debugger available called DDB. It allows setting of
|
|
breakpoints, single-stepping kernel functions, examining and changing
|
|
kernel variables, etc. However, it cannot access kernel source files,
|
|
and only has access to the global and static symbols, not to the full
|
|
debug information like <command>gdb</command> does.</para>
|
|
|
|
<para>To configure your kernel to include DDB, add the option line
|
|
|
|
<programlisting>options DDB</programlisting>
|
|
|
|
to your config file, and rebuild. (See <ulink
|
|
url="../handbook/index.html">The FreeBSD Handbook</ulink> for details on
|
|
configuring the FreeBSD kernel).</para>
|
|
|
|
<note>
|
|
<para>If you have an older version of the boot blocks, your
|
|
debugger symbols might not be loaded at all. Update the boot blocks;
|
|
the recent ones load the DDB symbols automagically.</para>
|
|
</note>
|
|
|
|
<para>Once your DDB kernel is running, there are several ways to enter
|
|
DDB. The first, and earliest way is to type the boot flag
|
|
<option>-d</option> right at the boot prompt. The kernel will start up
|
|
in debug mode and enter DDB prior to any device probing. Hence you can
|
|
even debug the device probe/attach functions.</para>
|
|
|
|
<para>The second scenario is to drop to the debugger once the
|
|
system has booted. There are two simple ways to accomplish
|
|
this. If you would like to break to the debugger from the
|
|
command prompt, simply type the command:</para>
|
|
|
|
<screen>&prompt.root; <userinput>sysctl debug.enter_debugger=ddb</userinput></screen>
|
|
|
|
<para>Alternatively, if you are at the system console, you may use
|
|
a hot-key on the keyboard. The default break-to-debugger
|
|
sequence is <keycombo action="simul"><keycap>Ctrl</keycap>
|
|
<keycap>Alt</keycap><keycap>ESC</keycap></keycombo>. For
|
|
syscons, this sequence can be remapped and some of the
|
|
distributed maps out there do this, so check to make sure you
|
|
know the right sequence to use. There is an option available
|
|
for serial consoles that allows the use of a serial line BREAK on the
|
|
console line to enter DDB (<literal>options BREAK_TO_DEBUGGER</literal>
|
|
in the kernel config file). It is not the default since there are a lot
|
|
of serial adapters around that gratuitously generate a BREAK
|
|
condition, for example when pulling the cable.</para>
|
|
|
|
<para>The third way is that any panic condition will branch to DDB if the
|
|
kernel is configured to use it. For this reason, it is not wise to
|
|
configure a kernel with DDB for a machine running unattended.</para>
|
|
|
|
<para>The DDB commands roughly resemble some <command>gdb</command>
|
|
commands. The first thing you probably need to do is to set a
|
|
breakpoint:</para>
|
|
|
|
<screen><userinput>b function-name</userinput>
|
|
<userinput>b address</userinput></screen>
|
|
|
|
<para>Numbers are taken hexadecimal by default, but to make them distinct
|
|
from symbol names; hexadecimal numbers starting with the letters
|
|
<literal>a-f</literal> need to be preceded with <literal>0x</literal>
|
|
(this is optional for other numbers). Simple expressions are allowed,
|
|
for example: <literal>function-name + 0x103</literal>.</para>
|
|
|
|
<para>To continue the operation of an interrupted kernel, simply
|
|
type:</para>
|
|
|
|
<screen><userinput>c</userinput></screen>
|
|
|
|
<para>To get a stack trace, use:</para>
|
|
|
|
<screen><userinput>trace</userinput></screen>
|
|
|
|
<note>
|
|
<para>Note that when entering DDB via a hot-key, the kernel is currently
|
|
servicing an interrupt, so the stack trace might be not of much use
|
|
to you.</para>
|
|
</note>
|
|
|
|
<para>If you want to remove a breakpoint, use</para>
|
|
|
|
|
|
<screen><userinput>del</userinput>
|
|
<userinput>del address-expression</userinput></screen>
|
|
|
|
<para>The first form will be accepted immediately after a breakpoint hit,
|
|
and deletes the current breakpoint. The second form can remove any
|
|
breakpoint, but you need to specify the exact address; this can be
|
|
obtained from:</para>
|
|
|
|
<screen><userinput>show b</userinput></screen>
|
|
|
|
<para>To single-step the kernel, try:</para>
|
|
|
|
<screen><userinput>s</userinput></screen>
|
|
|
|
<para>This will step into functions, but you can make DDB trace them until
|
|
the matching return statement is reached by:</para>
|
|
|
|
<screen><userinput>n</userinput></screen>
|
|
|
|
<note>
|
|
<para>This is different from <command>gdb</command>'s
|
|
<command>next</command> statement; it is like <command>gdb</command>'s
|
|
<command>finish</command>.</para>
|
|
</note>
|
|
|
|
<para>To examine data from memory, use (for example):
|
|
|
|
<screen><userinput>x/wx 0xf0133fe0,40</userinput>
|
|
<userinput>x/hd db_symtab_space</userinput>
|
|
<userinput>x/bc termbuf,10</userinput>
|
|
<userinput>x/s stringbuf</userinput></screen>
|
|
|
|
for word/halfword/byte access, and hexadecimal/decimal/character/ string
|
|
display. The number after the comma is the object count. To display
|
|
the next 0x10 items, simply use:</para>
|
|
|
|
<screen><userinput>x ,10</userinput></screen>
|
|
|
|
<para>Similarly, use
|
|
|
|
<screen><userinput>x/ia foofunc,10</userinput></screen>
|
|
|
|
to disassemble the first 0x10 instructions of
|
|
<function>foofunc</function>, and display them along with their offset
|
|
from the beginning of <function>foofunc</function>.</para>
|
|
|
|
<para>To modify memory, use the write command:</para>
|
|
|
|
<screen><userinput>w/b termbuf 0xa 0xb 0</userinput>
|
|
<userinput>w/w 0xf0010030 0 0</userinput></screen>
|
|
|
|
<para>The command modifier
|
|
(<literal>b</literal>/<literal>h</literal>/<literal>w</literal>)
|
|
specifies the size of the data to be written, the first following
|
|
expression is the address to write to and the remainder is interpreted
|
|
as data to write to successive memory locations.</para>
|
|
|
|
<para>If you need to know the current registers, use:</para>
|
|
|
|
<screen><userinput>show reg</userinput></screen>
|
|
|
|
<para>Alternatively, you can display a single register value by e.g.
|
|
|
|
<screen><userinput>p $eax</userinput></screen>
|
|
|
|
and modify it by:</para>
|
|
|
|
<screen><userinput>set $eax new-value</userinput></screen>
|
|
|
|
<para>Should you need to call some kernel functions from DDB, simply
|
|
say:</para>
|
|
|
|
<screen><userinput>call func(arg1, arg2, ...)</userinput></screen>
|
|
|
|
<para>The return value will be printed.</para>
|
|
|
|
<para>For a &man.ps.1; style summary of all running processes, use:</para>
|
|
|
|
<screen><userinput>ps</userinput></screen>
|
|
|
|
<para>Now you have examined why your kernel failed, and you wish to
|
|
reboot. Remember that, depending on the severity of previous
|
|
malfunctioning, not all parts of the kernel might still be working as
|
|
expected. Perform one of the following actions to shut down and reboot
|
|
your system:</para>
|
|
|
|
<screen><userinput>panic</userinput></screen>
|
|
|
|
<para>This will cause your kernel to dump core and reboot, so you can
|
|
later analyze the core on a higher level with <command>gdb</command>. This command
|
|
usually must be followed by another <command>continue</command>
|
|
statement.</para>
|
|
|
|
<screen><userinput>call boot(0)</userinput></screen>
|
|
|
|
<para>Which might be a good way to cleanly shut down the running system,
|
|
<function>sync()</function> all disks, and finally reboot. As long as
|
|
the disk and file system interfaces of the kernel are not damaged, this
|
|
might be a good way for an almost clean shutdown.</para>
|
|
|
|
<screen><userinput>call cpu_reset()</userinput></screen>
|
|
|
|
<para>This is the final way out of disaster and almost the same as hitting the
|
|
Big Red Button.</para>
|
|
|
|
<para>If you need a short command summary, simply type:</para>
|
|
|
|
<screen><userinput>help</userinput></screen>
|
|
|
|
<para>However, it is highly recommended to have a printed copy of the
|
|
&man.ddb.4; manual page ready for a debugging
|
|
session. Remember that it is hard to read the on-line manual while
|
|
single-stepping the kernel.</para>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>On-Line Kernel Debugging Using Remote GDB</title>
|
|
|
|
<para>This feature has been supported since FreeBSD 2.2, and it is
|
|
actually a very neat one.</para>
|
|
|
|
<para>GDB has already supported <emphasis>remote debugging</emphasis> for
|
|
a long time. This is done using a very simple protocol along a serial
|
|
line. Unlike the other methods described above, you will need two
|
|
machines for doing this. One is the host providing the debugging
|
|
environment, including all the sources, and a copy of the kernel binary
|
|
with all the symbols in it, and the other one is the target machine that
|
|
simply runs a similar copy of the very same kernel (but stripped of the
|
|
debugging information).</para>
|
|
|
|
<para>You should configure the kernel in question with <command>config
|
|
-g</command>, include <option>DDB</option> into the configuration, and
|
|
compile it as usual. This gives a large binary, due to the
|
|
debugging information. Copy this kernel to the target machine, strip
|
|
the debugging symbols off with <command>strip -x</command>, and boot it
|
|
using the <option>-d</option> boot option. Connect the serial line
|
|
of the target machine that has "flags 080" set on its sio device
|
|
to any serial line of the debugging host.
|
|
Now, on the debugging machine, go to the compile directory of the target
|
|
kernel, and start <command>gdb</command>:</para>
|
|
|
|
<screen>&prompt.user; <userinput>gdb -k kernel</userinput>
|
|
GDB is free software and you are welcome to distribute copies of it
|
|
under certain conditions; type "show copying" to see the conditions.
|
|
There is absolutely no warranty for GDB; type "show warranty" for details.
|
|
GDB 4.16 (i386-unknown-freebsd),
|
|
Copyright 1996 Free Software Foundation, Inc...
|
|
<prompt>(kgdb)</prompt> </screen>
|
|
|
|
<para>Initialize the remote debugging session (assuming the first serial
|
|
port is being used) by:</para>
|
|
|
|
<screen><prompt>(kgdb)</prompt> <userinput>target remote /dev/cuaa0</userinput></screen>
|
|
|
|
<para>Now, on the target host (the one that entered DDB right before even
|
|
starting the device probe), type:</para>
|
|
|
|
<screen>Debugger("Boot flags requested debugger")
|
|
Stopped at Debugger+0x35: movb $0, edata+0x51bc
|
|
<prompt>db></prompt> <userinput>gdb</userinput></screen>
|
|
|
|
<para>DDB will respond with:</para>
|
|
|
|
<screen>Next trap will enter GDB remote protocol mode</screen>
|
|
|
|
<para>Every time you type <command>gdb</command>, the mode will be toggled
|
|
between remote GDB and local DDB. In order to force a next trap
|
|
immediately, simply type <command>s</command> (step). Your hosting GDB
|
|
will now gain control over the target kernel:</para>
|
|
|
|
<screen>Remote debugging using /dev/cuaa0
|
|
Debugger (msg=0xf01b0383 "Boot flags requested debugger")
|
|
at ../../i386/i386/db_interface.c:257
|
|
<prompt>(kgdb)</prompt></screen>
|
|
|
|
<para>You can use this session almost as any other GDB session, including
|
|
full access to the source, running it in gud-mode inside an Emacs window
|
|
(which gives you an automatic source code display in another Emacs
|
|
window), etc.</para>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>Debugging Loadable Modules Using GDB</title>
|
|
|
|
<para>When debugging a panic that occurred within a module, or
|
|
using remote GDB against a machine that uses dynamic modules,
|
|
you need to tell GDB how to obtain symbol information for those
|
|
modules.</para>
|
|
|
|
<para>First, you need to build the module(s) with debugging
|
|
information:</para>
|
|
|
|
<screen>&prompt.root; <userinput>cd /sys/modules/linux</userinput>
|
|
&prompt.root; <userinput>make clean; make COPTS=-g</userinput></screen>
|
|
|
|
<para>If you are using remote GDB, you can run
|
|
<command>kldstat</command> on the target machine to find out
|
|
where the module was loaded:</para>
|
|
|
|
<screen>&prompt.root; <userinput>kldstat</userinput>
|
|
Id Refs Address Size Name
|
|
1 4 0xc0100000 1c1678 kernel
|
|
2 1 0xc0a9e000 6000 linprocfs.ko
|
|
3 1 0xc0ad7000 2000 warp_saver.ko
|
|
4 1 0xc0adc000 11000 linux.ko</screen>
|
|
|
|
<para>If you are debugging a crash dump, you will need to walk the
|
|
<literal>linker_files</literal> list, starting at
|
|
<literal>linker_files->tqh_first</literal> and following the
|
|
<literal>link.tqe_next</literal> pointers until you find the
|
|
entry with the <literal>filename</literal> you are looking for.
|
|
The <literal>address</literal> member of that entry is the load
|
|
address of the module.</para>
|
|
|
|
<para>Next, you need to find out the offset of the text section
|
|
within the module:</para>
|
|
|
|
<screen>&prompt.root; <userinput>objdump --section-headers /sys/modules/linux/linux.ko | grep text</userinput>
|
|
3 .rel.text 000016e0 000038e0 000038e0 000038e0 2**2
|
|
10 .text 00007f34 000062d0 000062d0 000062d0 2**2</screen>
|
|
|
|
<para>The one you want is the <literal>.text</literal> section,
|
|
section 10 in the above example. The fourth hexadecimal field
|
|
(sixth field overall) is the offset of the text section within
|
|
the file. Add this offset to the load address of the module to
|
|
obtain the relocation address for the module's code. In our
|
|
example, we get 0xc0adc000 + 0x62d0 = 0xc0ae22d0. Use the
|
|
<command>add-symbol-file</command> command in GDB to tell the
|
|
debugger about the module:</para>
|
|
|
|
<screen><prompt>(kgdb)</prompt> <userinput>add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0</userinput>
|
|
add symbol table from file "/sys/modules/linux/linux.ko" at text_addr = 0xc0ae22d0?
|
|
(y or n) <userinput>y</userinput>
|
|
Reading symbols from /sys/modules/linux/linux.ko...done.
|
|
<prompt>(kgdb)</prompt></screen>
|
|
|
|
<para>You should now have access to all the symbols in the
|
|
module.</para>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>Debugging a Console Driver</title>
|
|
|
|
<para>Since you need a console driver to run DDB on, things are more
|
|
complicated if the console driver itself is failing. You might remember
|
|
the use of a serial console (either with modified boot blocks, or by
|
|
specifying <option>-h</option> at the <prompt>Boot:</prompt> prompt),
|
|
and hook up a standard terminal onto your first serial port. DDB works
|
|
on any configured console driver, including a serial
|
|
console.</para>
|
|
</sect1>
|
|
</chapter>
|
|
|
|
<!--
|
|
Local Variables:
|
|
mode: sgml
|
|
sgml-declaration: "../chapter.decl"
|
|
sgml-indent-data: t
|
|
sgml-omittag: nil
|
|
sgml-always-quote-attributes: t
|
|
sgml-parent-document: ("../book.sgml" "part" "chapter")
|
|
End:
|
|
-->
|
|
|