Nuke whitespace before running over this chapter in an attempt to update

various examples so that this chapter is useful to new and unsuspecting
vict^H^H^H^Huser^H^H^H^Hkernel debuggers.

Submitted by:	echo "(setq-default show-trailing-whitespace t)" >> ~/.emacs
This commit is contained in:
Sean Chittenden 2003-10-17 21:16:23 +00:00
parent 2c9abf4694
commit 4bea63b48e
Notes: svn2git 2020-12-08 03:00:23 +00:00
svn path=/head/; revision=18474

View file

@ -6,12 +6,12 @@
<chapter id="kerneldebug">
<title>Kernel Debugging</title>
<para><emphasis>Contributed by &a.paul; and &a.joerg;</emphasis></para>
<sect1 id="kerneldebug-gdb">
<title>Debugging a Kernel Crash Dump with <command>gdb</command></title>
<para>Here are some instructions for getting kernel debugging
working on a crash dump. They assume that you have enough swap
space for a crash dump. Typically you want to
@ -38,16 +38,16 @@
be used only if you want a crash dump from a kernel that crashes during
booting.</para>
</note>
<note>
<para>In the following, the term <command>gdb</command> refers to
the debugger <command>gdb</command> run in <quote>kernel debug
mode</quote>. This can be accomplished by starting the
<command>gdb</command> with the option <option>-k</option>. In
kernel debug mode, <command>gdb</command> changes its prompt to
the debugger <command>gdb</command> run in <quote>kernel debug
mode</quote>. This can be accomplished by starting the
<command>gdb</command> with the option <option>-k</option>. In
kernel debug mode, <command>gdb</command> changes its prompt to
<prompt>(kgdb)</prompt>.</para>
</note>
<tip>
<para>If you are using FreeBSD 3 or earlier, you should make a stripped
copy of the debug kernel, rather than installing the large debug
@ -81,20 +81,20 @@
&prompt.root; <userinput>mount -a -t ufs</userinput> # so your filesystem for /var/crash is writable
&prompt.root; <userinput>savecore -N /kernel.panicked /var/crash</userinput>
&prompt.root; <userinput>exit</userinput> # ...to multi-user</screen>
<para>This instructs &man.savecore.8; to use another kernel for symbol
name extraction. It would otherwise default to the currently running
kernel and most likely not do anything at all since the crash dump and
the kernel symbols differ.</para>
<para>Now, after a crash dump, go to
<filename>/sys/compile/WHATEVER</filename> and run
<command>gdb <option>-k</option></command>. From <command>gdb</command> do:
<screen><userinput>symbol-file kernel.debug</userinput>
<userinput>exec-file /var/crash/kernel.0</userinput>
<userinput>core-file /var/crash/vmcore.0</userinput></screen>
and voila, you can debug the crash dump using the kernel sources just
like you can for any other program.</para>
@ -103,10 +103,10 @@
readability, and the lines are numbered for reference. Despite this, it
is a real-world error trace taken during the development of the pcvt
console driver.</para>
<screen> 1:Script started on Fri Dec 30 23:15:22 1994
2:&prompt.root; <userinput>cd /sys/compile/URIAH</userinput>
3:&prompt.root; <userinput>gdb -k kernel /var/crash/vmcore.1</userinput>
3:&prompt.root; <userinput>gdb -k kernel /var/crash/vmcore.1</userinput>
4:Reading symbol data from /usr/src/sys/compile/URIAH/kernel
...done.
5:IdlePTD 1f3000
@ -146,10 +146,10 @@
39:roc *) 0xf07c0c00) (../../i386/isa/pcvt/pcvt_drv.c line 403)
40:403 return ((*linesw[tp-&gt;t_line].l_open)(dev, tp));
41:<prompt>(kgdb)</prompt> <userinput>list</userinput>
42:398
42:398
43:399 tp-&gt;t_state |= TS_CARR_ON;
44:400 tp-&gt;t_cflag |= CLOCAL; /* cannot be a modem (:-) */
45:401
45:401
46:402 #if PCVT_NETBSD || (PCVT_FREEBSD >= 200)
47:403 return ((*linesw[tp-&gt;t_line].l_open)(dev, tp));
48:404 #else
@ -186,7 +186,7 @@
79:
80:Script done on Fri Dec 30 23:18:04 1994</screen>
<para>Comments to the above script:</para>
<variablelist>
<varlistentry>
<term>line 6:</term>
@ -198,7 +198,7 @@
page fault trap though.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>line 20:</term>
@ -207,7 +207,7 @@
in the stack trace.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>line 36:</term>
@ -221,7 +221,7 @@
bounds.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>line 52:</term>
@ -230,7 +230,7 @@
address.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>line 56:</term>
@ -244,10 +244,10 @@
</varlistentry>
</variablelist>
</sect1>
<sect1 id="kerneldebug-ddd">
<title>Debugging a Crash Dump with DDD</title>
<para>Examining a kernel crash dump with a graphical debugger like
<command>ddd</command> is also possible (you will need to install
the <filename role="package">devel/ddd</filename> port in order to use the
@ -256,21 +256,21 @@
normally. For example;</para>
<screen>&prompt.root; <userinput>ddd -k /var/crash/kernel.0 /var/crash/vmcore.0</userinput></screen>
<para>You should then be able to go about looking at the crash dump using
<command>ddd</command>'s graphical interface.</para>
</sect1>
<sect1 id="kerneldebug-post-mortem">
<title>Post-Mortem Analysis of a Dump</title>
<para>What do you do if a kernel dumped core but you did not expect it,
and it is therefore not compiled using <command>config -g</command>? Not
everything is lost here. Do not panic!</para>
<para>Of course, you still need to enable crash dumps. See above for the
options you have to specify in order to do this.</para>
<para>Go to your kernel config directory
(<filename>/usr/src/sys/<replaceable>arch</replaceable>/conf</filename>)
and edit your configuration file. Uncomment (or add, if it does not
@ -298,10 +298,10 @@
<para>All this is not guaranteed to work, but it will do it fine in most
cases.</para>
</sect1>
<sect1 id="kerneldebug-online-ddb">
<title>On-Line Kernel Debugging Using DDB</title>
<para>While <command>gdb <option>-k</option></command> as an off-line debugger provides a very
high level of user interface, there are some things it cannot do. The
most important ones being breakpointing and single-stepping kernel
@ -315,7 +315,7 @@
debug information like <command>gdb</command> does.</para>
<para>To configure your kernel to include DDB, add the option line
<programlisting>options DDB</programlisting>
to your config file, and rebuild. (See <ulink
@ -327,7 +327,7 @@
debugger symbols might not be loaded at all. Update the boot blocks;
the recent ones load the DDB symbols automatically.</para>
</note>
<para>Once your DDB kernel is running, there are several ways to enter
DDB. The first, and earliest way is to type the boot flag
<option>-d</option> right at the boot prompt. The kernel will start up
@ -361,10 +361,10 @@
<para>The DDB commands roughly resemble some <command>gdb</command>
commands. The first thing you probably need to do is to set a
breakpoint:</para>
<screen><userinput>b function-name</userinput>
<userinput>b address</userinput></screen>
<para>Numbers are taken hexadecimal by default, but to make them distinct
from symbol names; hexadecimal numbers starting with the letters
<literal>a-f</literal> need to be preceded with <literal>0x</literal>
@ -373,39 +373,39 @@
<para>To continue the operation of an interrupted kernel, simply
type:</para>
<screen><userinput>c</userinput></screen>
<para>To get a stack trace, use:</para>
<screen><userinput>trace</userinput></screen>
<note>
<para>Note that when entering DDB via a hot-key, the kernel is currently
servicing an interrupt, so the stack trace might be not of much use
to you.</para>
</note>
<para>If you want to remove a breakpoint, use</para>
<screen><userinput>del</userinput>
<userinput>del address-expression</userinput></screen>
<para>The first form will be accepted immediately after a breakpoint hit,
and deletes the current breakpoint. The second form can remove any
breakpoint, but you need to specify the exact address; this can be
obtained from:</para>
<screen><userinput>show b</userinput></screen>
<para>To single-step the kernel, try:</para>
<screen><userinput>s</userinput></screen>
<para>This will step into functions, but you can make DDB trace them until
the matching return statement is reached by:</para>
<screen><userinput>n</userinput></screen>
<note>
@ -413,33 +413,33 @@
<command>next</command> statement; it is like <command>gdb</command>'s
<command>finish</command>.</para>
</note>
<para>To examine data from memory, use (for example):
<screen><userinput>x/wx 0xf0133fe0,40</userinput>
<userinput>x/hd db_symtab_space</userinput>
<userinput>x/bc termbuf,10</userinput>
<userinput>x/s stringbuf</userinput></screen>
for word/halfword/byte access, and hexadecimal/decimal/character/ string
display. The number after the comma is the object count. To display
the next 0x10 items, simply use:</para>
<screen><userinput>x ,10</userinput></screen>
<para>Similarly, use
<screen><userinput>x/ia foofunc,10</userinput></screen>
to disassemble the first 0x10 instructions of
<function>foofunc</function>, and display them along with their offset
from the beginning of <function>foofunc</function>.</para>
<para>To modify memory, use the write command:</para>
<screen><userinput>w/b termbuf 0xa 0xb 0</userinput>
<userinput>w/w 0xf0010030 0 0</userinput></screen>
<para>The command modifier
(<literal>b</literal>/<literal>h</literal>/<literal>w</literal>)
specifies the size of the data to be written, the first following
@ -457,18 +457,18 @@
and modify it by:</para>
<screen><userinput>set $eax new-value</userinput></screen>
<para>Should you need to call some kernel functions from DDB, simply
say:</para>
<screen><userinput>call func(arg1, arg2, ...)</userinput></screen>
<para>The return value will be printed.</para>
<para>For a &man.ps.1; style summary of all running processes, use:</para>
<screen><userinput>ps</userinput></screen>
<para>Now you have examined why your kernel failed, and you wish to
reboot. Remember that, depending on the severity of previous
malfunctioning, not all parts of the kernel might still be working as
@ -476,40 +476,40 @@
your system:</para>
<screen><userinput>panic</userinput></screen>
<para>This will cause your kernel to dump core and reboot, so you can
later analyze the core on a higher level with <command>gdb</command>. This command
usually must be followed by another <command>continue</command>
statement.</para>
<screen><userinput>call boot(0)</userinput></screen>
<para>Which might be a good way to cleanly shut down the running system,
<function>sync()</function> all disks, and finally reboot. As long as
the disk and filesystem interfaces of the kernel are not damaged, this
might be a good way for an almost clean shutdown.</para>
<screen><userinput>call cpu_reset()</userinput></screen>
<para>This is the final way out of disaster and almost the same as hitting the
Big Red Button.</para>
<para>If you need a short command summary, simply type:</para>
<screen><userinput>help</userinput></screen>
<para>However, it is highly recommended to have a printed copy of the
&man.ddb.4; manual page ready for a debugging
session. Remember that it is hard to read the on-line manual while
single-stepping the kernel.</para>
</sect1>
<sect1 id="kerneldebug-online-gdb">
<title>On-Line Kernel Debugging Using Remote GDB</title>
<para>This feature has been supported since FreeBSD 2.2, and it is
actually a very neat one.</para>
<para>GDB has already supported <emphasis>remote debugging</emphasis> for
a long time. This is done using a very simple protocol along a serial
line. Unlike the other methods described above, you will need two
@ -537,42 +537,42 @@ There is absolutely no warranty for GDB; type "show warranty" for details.
GDB 4.16 (i386-unknown-freebsd),
Copyright 1996 Free Software Foundation, Inc...
<prompt>(kgdb)</prompt> </screen>
<para>Initialize the remote debugging session (assuming the first serial
port is being used) by:</para>
<screen><prompt>(kgdb)</prompt> <userinput>target remote /dev/cuaa0</userinput></screen>
<para>Now, on the target host (the one that entered DDB right before even
starting the device probe), type:</para>
<screen>Debugger("Boot flags requested debugger")
Stopped at Debugger+0x35: movb $0, edata+0x51bc
<prompt>db&gt;</prompt> <userinput>gdb</userinput></screen>
<para>DDB will respond with:</para>
<screen>Next trap will enter GDB remote protocol mode</screen>
<para>Every time you type <command>gdb</command>, the mode will be toggled
between remote GDB and local DDB. In order to force a next trap
immediately, simply type <command>s</command> (step). Your hosting GDB
will now gain control over the target kernel:</para>
<screen>Remote debugging using /dev/cuaa0
Debugger (msg=0xf01b0383 "Boot flags requested debugger")
at ../../i386/i386/db_interface.c:257
<prompt>(kgdb)</prompt></screen>
<para>You can use this session almost as any other GDB session, including
full access to the source, running it in gud-mode inside an Emacs window
(which gives you an automatic source code display in another Emacs
window), etc.</para>
</sect1>
<sect1 id="kerneldebug-kld">
<title>Debugging Loadable Modules Using GDB</title>
<para>When debugging a panic that occurred within a module, or
using remote GDB against a machine that uses dynamic modules,
you need to tell GDB how to obtain symbol information for those
@ -583,7 +583,7 @@ Debugger (msg=0xf01b0383 "Boot flags requested debugger")
<screen>&prompt.root; <userinput>cd /sys/modules/linux</userinput>
&prompt.root; <userinput>make clean; make COPTS=-g</userinput></screen>
<para>If you are using remote GDB, you can run
<command>kldstat</command> on the target machine to find out
where the module was loaded:</para>
@ -602,7 +602,7 @@ Id Refs Address Size Name
entry with the <literal>filename</literal> you are looking for.
The <literal>address</literal> member of that entry is the load
address of the module.</para>
<para>Next, you need to find out the offset of the text section
within the module:</para>
@ -624,11 +624,11 @@ add symbol table from file "/sys/modules/linux/linux.ko" at text_addr = 0xc0ae22
(y or n) <userinput>y</userinput>
Reading symbols from /sys/modules/linux/linux.ko...done.
<prompt>(kgdb)</prompt></screen>
<para>You should now have access to all the symbols in the
module.</para>
</sect1>
<sect1 id="kerneldebug-console">
<title>Debugging a Console Driver</title>
@ -642,7 +642,7 @@ Reading symbols from /sys/modules/linux/linux.ko...done.
</sect1>
</chapter>
<!--
<!--
Local Variables:
mode: sgml
sgml-declaration: "../chapter.decl"