Nuke whitespace before running over this chapter in an attempt to update
various examples so that this chapter is useful to new and unsuspecting vict^H^H^H^Huser^H^H^H^Hkernel debuggers. Submitted by: echo "(setq-default show-trailing-whitespace t)" >> ~/.emacs
This commit is contained in:
parent
2c9abf4694
commit
4bea63b48e
Notes:
svn2git
2020-12-08 03:00:23 +00:00
svn path=/head/; revision=18474
1 changed files with 85 additions and 85 deletions
|
@ -6,12 +6,12 @@
|
|||
|
||||
<chapter id="kerneldebug">
|
||||
<title>Kernel Debugging</title>
|
||||
|
||||
|
||||
<para><emphasis>Contributed by &a.paul; and &a.joerg;</emphasis></para>
|
||||
|
||||
|
||||
<sect1 id="kerneldebug-gdb">
|
||||
<title>Debugging a Kernel Crash Dump with <command>gdb</command></title>
|
||||
|
||||
|
||||
<para>Here are some instructions for getting kernel debugging
|
||||
working on a crash dump. They assume that you have enough swap
|
||||
space for a crash dump. Typically you want to
|
||||
|
@ -38,16 +38,16 @@
|
|||
be used only if you want a crash dump from a kernel that crashes during
|
||||
booting.</para>
|
||||
</note>
|
||||
|
||||
|
||||
<note>
|
||||
<para>In the following, the term <command>gdb</command> refers to
|
||||
the debugger <command>gdb</command> run in <quote>kernel debug
|
||||
mode</quote>. This can be accomplished by starting the
|
||||
<command>gdb</command> with the option <option>-k</option>. In
|
||||
kernel debug mode, <command>gdb</command> changes its prompt to
|
||||
the debugger <command>gdb</command> run in <quote>kernel debug
|
||||
mode</quote>. This can be accomplished by starting the
|
||||
<command>gdb</command> with the option <option>-k</option>. In
|
||||
kernel debug mode, <command>gdb</command> changes its prompt to
|
||||
<prompt>(kgdb)</prompt>.</para>
|
||||
</note>
|
||||
|
||||
|
||||
<tip>
|
||||
<para>If you are using FreeBSD 3 or earlier, you should make a stripped
|
||||
copy of the debug kernel, rather than installing the large debug
|
||||
|
@ -81,20 +81,20 @@
|
|||
&prompt.root; <userinput>mount -a -t ufs</userinput> # so your filesystem for /var/crash is writable
|
||||
&prompt.root; <userinput>savecore -N /kernel.panicked /var/crash</userinput>
|
||||
&prompt.root; <userinput>exit</userinput> # ...to multi-user</screen>
|
||||
|
||||
|
||||
<para>This instructs &man.savecore.8; to use another kernel for symbol
|
||||
name extraction. It would otherwise default to the currently running
|
||||
kernel and most likely not do anything at all since the crash dump and
|
||||
the kernel symbols differ.</para>
|
||||
|
||||
|
||||
<para>Now, after a crash dump, go to
|
||||
<filename>/sys/compile/WHATEVER</filename> and run
|
||||
<command>gdb <option>-k</option></command>. From <command>gdb</command> do:
|
||||
|
||||
|
||||
<screen><userinput>symbol-file kernel.debug</userinput>
|
||||
<userinput>exec-file /var/crash/kernel.0</userinput>
|
||||
<userinput>core-file /var/crash/vmcore.0</userinput></screen>
|
||||
|
||||
|
||||
and voila, you can debug the crash dump using the kernel sources just
|
||||
like you can for any other program.</para>
|
||||
|
||||
|
@ -103,10 +103,10 @@
|
|||
readability, and the lines are numbered for reference. Despite this, it
|
||||
is a real-world error trace taken during the development of the pcvt
|
||||
console driver.</para>
|
||||
|
||||
|
||||
<screen> 1:Script started on Fri Dec 30 23:15:22 1994
|
||||
2:&prompt.root; <userinput>cd /sys/compile/URIAH</userinput>
|
||||
3:&prompt.root; <userinput>gdb -k kernel /var/crash/vmcore.1</userinput>
|
||||
3:&prompt.root; <userinput>gdb -k kernel /var/crash/vmcore.1</userinput>
|
||||
4:Reading symbol data from /usr/src/sys/compile/URIAH/kernel
|
||||
...done.
|
||||
5:IdlePTD 1f3000
|
||||
|
@ -146,10 +146,10 @@
|
|||
39:roc *) 0xf07c0c00) (../../i386/isa/pcvt/pcvt_drv.c line 403)
|
||||
40:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
|
||||
41:<prompt>(kgdb)</prompt> <userinput>list</userinput>
|
||||
42:398
|
||||
42:398
|
||||
43:399 tp->t_state |= TS_CARR_ON;
|
||||
44:400 tp->t_cflag |= CLOCAL; /* cannot be a modem (:-) */
|
||||
45:401
|
||||
45:401
|
||||
46:402 #if PCVT_NETBSD || (PCVT_FREEBSD >= 200)
|
||||
47:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
|
||||
48:404 #else
|
||||
|
@ -186,7 +186,7 @@
|
|||
79:
|
||||
80:Script done on Fri Dec 30 23:18:04 1994</screen>
|
||||
<para>Comments to the above script:</para>
|
||||
|
||||
|
||||
<variablelist>
|
||||
<varlistentry>
|
||||
<term>line 6:</term>
|
||||
|
@ -198,7 +198,7 @@
|
|||
page fault trap though.</para>
|
||||
</listitem>
|
||||
</varlistentry>
|
||||
|
||||
|
||||
<varlistentry>
|
||||
<term>line 20:</term>
|
||||
|
||||
|
@ -207,7 +207,7 @@
|
|||
in the stack trace.</para>
|
||||
</listitem>
|
||||
</varlistentry>
|
||||
|
||||
|
||||
<varlistentry>
|
||||
<term>line 36:</term>
|
||||
|
||||
|
@ -221,7 +221,7 @@
|
|||
bounds.</para>
|
||||
</listitem>
|
||||
</varlistentry>
|
||||
|
||||
|
||||
<varlistentry>
|
||||
<term>line 52:</term>
|
||||
|
||||
|
@ -230,7 +230,7 @@
|
|||
address.</para>
|
||||
</listitem>
|
||||
</varlistentry>
|
||||
|
||||
|
||||
<varlistentry>
|
||||
<term>line 56:</term>
|
||||
|
||||
|
@ -244,10 +244,10 @@
|
|||
</varlistentry>
|
||||
</variablelist>
|
||||
</sect1>
|
||||
|
||||
|
||||
<sect1 id="kerneldebug-ddd">
|
||||
<title>Debugging a Crash Dump with DDD</title>
|
||||
|
||||
|
||||
<para>Examining a kernel crash dump with a graphical debugger like
|
||||
<command>ddd</command> is also possible (you will need to install
|
||||
the <filename role="package">devel/ddd</filename> port in order to use the
|
||||
|
@ -256,21 +256,21 @@
|
|||
normally. For example;</para>
|
||||
|
||||
<screen>&prompt.root; <userinput>ddd -k /var/crash/kernel.0 /var/crash/vmcore.0</userinput></screen>
|
||||
|
||||
|
||||
<para>You should then be able to go about looking at the crash dump using
|
||||
<command>ddd</command>'s graphical interface.</para>
|
||||
</sect1>
|
||||
|
||||
|
||||
<sect1 id="kerneldebug-post-mortem">
|
||||
<title>Post-Mortem Analysis of a Dump</title>
|
||||
|
||||
|
||||
<para>What do you do if a kernel dumped core but you did not expect it,
|
||||
and it is therefore not compiled using <command>config -g</command>? Not
|
||||
everything is lost here. Do not panic!</para>
|
||||
|
||||
<para>Of course, you still need to enable crash dumps. See above for the
|
||||
options you have to specify in order to do this.</para>
|
||||
|
||||
|
||||
<para>Go to your kernel config directory
|
||||
(<filename>/usr/src/sys/<replaceable>arch</replaceable>/conf</filename>)
|
||||
and edit your configuration file. Uncomment (or add, if it does not
|
||||
|
@ -298,10 +298,10 @@
|
|||
<para>All this is not guaranteed to work, but it will do it fine in most
|
||||
cases.</para>
|
||||
</sect1>
|
||||
|
||||
|
||||
<sect1 id="kerneldebug-online-ddb">
|
||||
<title>On-Line Kernel Debugging Using DDB</title>
|
||||
|
||||
|
||||
<para>While <command>gdb <option>-k</option></command> as an off-line debugger provides a very
|
||||
high level of user interface, there are some things it cannot do. The
|
||||
most important ones being breakpointing and single-stepping kernel
|
||||
|
@ -315,7 +315,7 @@
|
|||
debug information like <command>gdb</command> does.</para>
|
||||
|
||||
<para>To configure your kernel to include DDB, add the option line
|
||||
|
||||
|
||||
<programlisting>options DDB</programlisting>
|
||||
|
||||
to your config file, and rebuild. (See <ulink
|
||||
|
@ -327,7 +327,7 @@
|
|||
debugger symbols might not be loaded at all. Update the boot blocks;
|
||||
the recent ones load the DDB symbols automatically.</para>
|
||||
</note>
|
||||
|
||||
|
||||
<para>Once your DDB kernel is running, there are several ways to enter
|
||||
DDB. The first, and earliest way is to type the boot flag
|
||||
<option>-d</option> right at the boot prompt. The kernel will start up
|
||||
|
@ -361,10 +361,10 @@
|
|||
<para>The DDB commands roughly resemble some <command>gdb</command>
|
||||
commands. The first thing you probably need to do is to set a
|
||||
breakpoint:</para>
|
||||
|
||||
|
||||
<screen><userinput>b function-name</userinput>
|
||||
<userinput>b address</userinput></screen>
|
||||
|
||||
|
||||
<para>Numbers are taken hexadecimal by default, but to make them distinct
|
||||
from symbol names; hexadecimal numbers starting with the letters
|
||||
<literal>a-f</literal> need to be preceded with <literal>0x</literal>
|
||||
|
@ -373,39 +373,39 @@
|
|||
|
||||
<para>To continue the operation of an interrupted kernel, simply
|
||||
type:</para>
|
||||
|
||||
|
||||
<screen><userinput>c</userinput></screen>
|
||||
|
||||
|
||||
<para>To get a stack trace, use:</para>
|
||||
|
||||
|
||||
<screen><userinput>trace</userinput></screen>
|
||||
|
||||
|
||||
<note>
|
||||
<para>Note that when entering DDB via a hot-key, the kernel is currently
|
||||
servicing an interrupt, so the stack trace might be not of much use
|
||||
to you.</para>
|
||||
</note>
|
||||
|
||||
|
||||
<para>If you want to remove a breakpoint, use</para>
|
||||
|
||||
|
||||
|
||||
|
||||
<screen><userinput>del</userinput>
|
||||
<userinput>del address-expression</userinput></screen>
|
||||
|
||||
|
||||
<para>The first form will be accepted immediately after a breakpoint hit,
|
||||
and deletes the current breakpoint. The second form can remove any
|
||||
breakpoint, but you need to specify the exact address; this can be
|
||||
obtained from:</para>
|
||||
|
||||
|
||||
<screen><userinput>show b</userinput></screen>
|
||||
|
||||
|
||||
<para>To single-step the kernel, try:</para>
|
||||
|
||||
|
||||
<screen><userinput>s</userinput></screen>
|
||||
|
||||
|
||||
<para>This will step into functions, but you can make DDB trace them until
|
||||
the matching return statement is reached by:</para>
|
||||
|
||||
|
||||
<screen><userinput>n</userinput></screen>
|
||||
|
||||
<note>
|
||||
|
@ -413,33 +413,33 @@
|
|||
<command>next</command> statement; it is like <command>gdb</command>'s
|
||||
<command>finish</command>.</para>
|
||||
</note>
|
||||
|
||||
|
||||
<para>To examine data from memory, use (for example):
|
||||
|
||||
|
||||
<screen><userinput>x/wx 0xf0133fe0,40</userinput>
|
||||
<userinput>x/hd db_symtab_space</userinput>
|
||||
<userinput>x/bc termbuf,10</userinput>
|
||||
<userinput>x/s stringbuf</userinput></screen>
|
||||
|
||||
|
||||
for word/halfword/byte access, and hexadecimal/decimal/character/ string
|
||||
display. The number after the comma is the object count. To display
|
||||
the next 0x10 items, simply use:</para>
|
||||
|
||||
|
||||
<screen><userinput>x ,10</userinput></screen>
|
||||
|
||||
|
||||
<para>Similarly, use
|
||||
|
||||
<screen><userinput>x/ia foofunc,10</userinput></screen>
|
||||
|
||||
|
||||
to disassemble the first 0x10 instructions of
|
||||
<function>foofunc</function>, and display them along with their offset
|
||||
from the beginning of <function>foofunc</function>.</para>
|
||||
|
||||
<para>To modify memory, use the write command:</para>
|
||||
|
||||
|
||||
<screen><userinput>w/b termbuf 0xa 0xb 0</userinput>
|
||||
<userinput>w/w 0xf0010030 0 0</userinput></screen>
|
||||
|
||||
|
||||
<para>The command modifier
|
||||
(<literal>b</literal>/<literal>h</literal>/<literal>w</literal>)
|
||||
specifies the size of the data to be written, the first following
|
||||
|
@ -457,18 +457,18 @@
|
|||
and modify it by:</para>
|
||||
|
||||
<screen><userinput>set $eax new-value</userinput></screen>
|
||||
|
||||
|
||||
<para>Should you need to call some kernel functions from DDB, simply
|
||||
say:</para>
|
||||
|
||||
<screen><userinput>call func(arg1, arg2, ...)</userinput></screen>
|
||||
|
||||
|
||||
<para>The return value will be printed.</para>
|
||||
|
||||
|
||||
<para>For a &man.ps.1; style summary of all running processes, use:</para>
|
||||
|
||||
|
||||
<screen><userinput>ps</userinput></screen>
|
||||
|
||||
|
||||
<para>Now you have examined why your kernel failed, and you wish to
|
||||
reboot. Remember that, depending on the severity of previous
|
||||
malfunctioning, not all parts of the kernel might still be working as
|
||||
|
@ -476,40 +476,40 @@
|
|||
your system:</para>
|
||||
|
||||
<screen><userinput>panic</userinput></screen>
|
||||
|
||||
|
||||
<para>This will cause your kernel to dump core and reboot, so you can
|
||||
later analyze the core on a higher level with <command>gdb</command>. This command
|
||||
usually must be followed by another <command>continue</command>
|
||||
statement.</para>
|
||||
|
||||
|
||||
<screen><userinput>call boot(0)</userinput></screen>
|
||||
|
||||
|
||||
<para>Which might be a good way to cleanly shut down the running system,
|
||||
<function>sync()</function> all disks, and finally reboot. As long as
|
||||
the disk and filesystem interfaces of the kernel are not damaged, this
|
||||
might be a good way for an almost clean shutdown.</para>
|
||||
|
||||
|
||||
<screen><userinput>call cpu_reset()</userinput></screen>
|
||||
|
||||
|
||||
<para>This is the final way out of disaster and almost the same as hitting the
|
||||
Big Red Button.</para>
|
||||
|
||||
<para>If you need a short command summary, simply type:</para>
|
||||
|
||||
|
||||
<screen><userinput>help</userinput></screen>
|
||||
|
||||
|
||||
<para>However, it is highly recommended to have a printed copy of the
|
||||
&man.ddb.4; manual page ready for a debugging
|
||||
session. Remember that it is hard to read the on-line manual while
|
||||
single-stepping the kernel.</para>
|
||||
</sect1>
|
||||
|
||||
|
||||
<sect1 id="kerneldebug-online-gdb">
|
||||
<title>On-Line Kernel Debugging Using Remote GDB</title>
|
||||
|
||||
|
||||
<para>This feature has been supported since FreeBSD 2.2, and it is
|
||||
actually a very neat one.</para>
|
||||
|
||||
|
||||
<para>GDB has already supported <emphasis>remote debugging</emphasis> for
|
||||
a long time. This is done using a very simple protocol along a serial
|
||||
line. Unlike the other methods described above, you will need two
|
||||
|
@ -537,42 +537,42 @@ There is absolutely no warranty for GDB; type "show warranty" for details.
|
|||
GDB 4.16 (i386-unknown-freebsd),
|
||||
Copyright 1996 Free Software Foundation, Inc...
|
||||
<prompt>(kgdb)</prompt> </screen>
|
||||
|
||||
|
||||
<para>Initialize the remote debugging session (assuming the first serial
|
||||
port is being used) by:</para>
|
||||
|
||||
<screen><prompt>(kgdb)</prompt> <userinput>target remote /dev/cuaa0</userinput></screen>
|
||||
|
||||
|
||||
<para>Now, on the target host (the one that entered DDB right before even
|
||||
starting the device probe), type:</para>
|
||||
|
||||
|
||||
<screen>Debugger("Boot flags requested debugger")
|
||||
Stopped at Debugger+0x35: movb $0, edata+0x51bc
|
||||
<prompt>db></prompt> <userinput>gdb</userinput></screen>
|
||||
|
||||
|
||||
<para>DDB will respond with:</para>
|
||||
|
||||
|
||||
<screen>Next trap will enter GDB remote protocol mode</screen>
|
||||
|
||||
|
||||
<para>Every time you type <command>gdb</command>, the mode will be toggled
|
||||
between remote GDB and local DDB. In order to force a next trap
|
||||
immediately, simply type <command>s</command> (step). Your hosting GDB
|
||||
will now gain control over the target kernel:</para>
|
||||
|
||||
|
||||
<screen>Remote debugging using /dev/cuaa0
|
||||
Debugger (msg=0xf01b0383 "Boot flags requested debugger")
|
||||
at ../../i386/i386/db_interface.c:257
|
||||
<prompt>(kgdb)</prompt></screen>
|
||||
|
||||
|
||||
<para>You can use this session almost as any other GDB session, including
|
||||
full access to the source, running it in gud-mode inside an Emacs window
|
||||
(which gives you an automatic source code display in another Emacs
|
||||
window), etc.</para>
|
||||
</sect1>
|
||||
|
||||
|
||||
<sect1 id="kerneldebug-kld">
|
||||
<title>Debugging Loadable Modules Using GDB</title>
|
||||
|
||||
|
||||
<para>When debugging a panic that occurred within a module, or
|
||||
using remote GDB against a machine that uses dynamic modules,
|
||||
you need to tell GDB how to obtain symbol information for those
|
||||
|
@ -583,7 +583,7 @@ Debugger (msg=0xf01b0383 "Boot flags requested debugger")
|
|||
|
||||
<screen>&prompt.root; <userinput>cd /sys/modules/linux</userinput>
|
||||
&prompt.root; <userinput>make clean; make COPTS=-g</userinput></screen>
|
||||
|
||||
|
||||
<para>If you are using remote GDB, you can run
|
||||
<command>kldstat</command> on the target machine to find out
|
||||
where the module was loaded:</para>
|
||||
|
@ -602,7 +602,7 @@ Id Refs Address Size Name
|
|||
entry with the <literal>filename</literal> you are looking for.
|
||||
The <literal>address</literal> member of that entry is the load
|
||||
address of the module.</para>
|
||||
|
||||
|
||||
<para>Next, you need to find out the offset of the text section
|
||||
within the module:</para>
|
||||
|
||||
|
@ -624,11 +624,11 @@ add symbol table from file "/sys/modules/linux/linux.ko" at text_addr = 0xc0ae22
|
|||
(y or n) <userinput>y</userinput>
|
||||
Reading symbols from /sys/modules/linux/linux.ko...done.
|
||||
<prompt>(kgdb)</prompt></screen>
|
||||
|
||||
|
||||
<para>You should now have access to all the symbols in the
|
||||
module.</para>
|
||||
</sect1>
|
||||
|
||||
|
||||
<sect1 id="kerneldebug-console">
|
||||
<title>Debugging a Console Driver</title>
|
||||
|
||||
|
@ -642,7 +642,7 @@ Reading symbols from /sys/modules/linux/linux.ko...done.
|
|||
</sect1>
|
||||
</chapter>
|
||||
|
||||
<!--
|
||||
<!--
|
||||
Local Variables:
|
||||
mode: sgml
|
||||
sgml-declaration: "../chapter.decl"
|
||||
|
|
Loading…
Reference in a new issue