<?xml version="1.0" encoding="iso-8859-1"?>
<!--
Recently I suggested to myself that this should become a profiling
and debugging chapter, which covers things like ktrace(1) and
using other debugging (like -x in shell scripts). But then I
realized that, over time and while DTrace becomes better supported,
that might make this chapter too large.
-->
<!--
The FreeBSD Documentation Project
$FreeBSD$
-->
<chapter id="dtrace">
<chapterinfo>
<authorgroup>
<author>
<firstname>Tom</firstname>
<surname>Rhodes</surname>
<contrib>Written by </contrib>
</author>
</authorgroup>
</chapterinfo>
<title>&dtrace;</title>
<sect1 id="dtrace-synopsis">
<title>Synopsis</title>
<indexterm><primary>&dtrace;</primary></indexterm>
<indexterm>
<primary>&dtrace; support</primary>
<see>&dtrace;</see>
</indexterm>
<para>&dtrace;, also known as Dynamic Tracing, was developed by
&sun; as a tool for locating performance bottlenecks in
production and pre-production systems. It is not, in any way,
a debugging tool, but a tool for real time system analysis to
locate performance and other issues.</para>
<para>&dtrace; is a remarkable profiling tool, with an impressive
array of features for diagnosing system issues. It may also
be used to run pre-written scripts to take advantage of its
capabilities. Users may even author their own utilities using
the &dtrace; D Language, allowing them to customize their
profiling based on specific needs.</para>
<para>After reading this chapter, you will know:</para>
<itemizedlist>
<listitem>
<para>What &dtrace; is and what features it provides.</para>
</listitem>
<listitem>
<para>Differences between the &solaris; &dtrace;
implementation and the one provided by &os;.</para>
</listitem>
<listitem>
<para>How to enable and use &dtrace; on &os;.</para>
</listitem>
</itemizedlist>
<para>Before reading this chapter, you should:</para>
<itemizedlist>
<listitem>
<para>Understand &unix; and &os; basics
(<xref linkend="basics"/>).</para>
</listitem>
<listitem>
<para>Be familiar with
the basics of kernel configuration/compilation
(<xref linkend="kernelconfig"/>).</para>
</listitem>
<listitem>
<para>Have some familiarity with security and how it
pertains to &os; (<xref linkend="security"/>).</para>
</listitem>
<listitem>
<para>Understand how to obtain and rebuild the &os; sources
(<xref linkend="updating-upgrading"/>).</para>
</listitem>
</itemizedlist>
<!--
Temporary warning to avoid listing experimental versions
and production versions of FreeBSD with this technology.
-->
<warning>
<para>This feature is considered experimental. Some options
may be lacking in functionality, other parts may not work
at all. In time, this feature will be considered production
ready and this documentation will be altered to fit that
situation.</para>
</warning>
</sect1>
<sect1 id="dtrace-implementation">
<title>Implementation Differences</title>
<para>While the &dtrace; in &os; is very similar to that found
in &solaris;, differences exist that should be explained before
continuing. The primary difference users will notice is that
on &os;, &dtrace; needs to be specifically enabled. There are
kernel options and modules which must be enabled for &dtrace; to
work properly. These will be explained later.</para>
<para>There is a <literal>DDB_CTF</literal> kernel option which
is used to enable support for loading the <acronym>CTF</acronym>
data from kernel modules and the kernel itself.
<acronym>CTF</acronym> is the &solaris; Compact C Type Format
which encapsulates a reduced form of debugging information
similar to <acronym>DWARF</acronym> and the venerable stabs.
This <acronym>CTF</acronym> data is added to the binaries by the
<command>ctfconvert</command> and <command>ctfmerge</command>
build tools. The <command>ctfconvert</command> utility parses
<acronym>DWARF</acronym> <acronym>ELF</acronym> debug sections
created by the compiler and <command>ctfmerge</command> merges
<acronym>CTF</acronym> <acronym>ELF</acronym> sections from
objects into either executables or shared libraries. More on
how to enable this for the kernel and &os; build is
forthcoming.</para>
<para>Some different providers exist for &os; than for &solaris;.
Most notable is the <literal>dtmalloc</literal> provider, which
allows tracing <function>malloc()</function> by type in the
&os; kernel.</para>
<para>Only <username>root</username> may use &dtrace; on &os;.
This is related to security differences, &solaris; has a few
low level security checks which do not yet exist in &os;. As
such, the <devicename>/dev/dtrace/dtrace</devicename> is
strictly limited to <username>root</username> users only.</para>
<para>Finally, the &dtrace; software falls under &sun;'s
<acronym>CDDL</acronym> license. The <literal>Common
Development and Distribution License</literal> comes with &os;,
see the
<filename>/usr/src/cddl/contrib/opensolaris/OPENSOLARIS.LICENSE</filename>
or view it online at
<ulink
url="http://www.opensolaris.org/os/licensing"></ulink>.</para>
<para>This license means that a &os; kernel with the &dtrace;
options is still <acronym>BSD</acronym> licensed; however
the <acronym>CDDL</acronym> kicks in when the modules are
distributed in binary form, or the binaries are loaded.</para>
</sect1>
<sect1 id="dtrace-enable">
<title>Enabling &dtrace; Support</title>
<para>To enable support for &dtrace;, add the following lines to
the kernel configuration file:</para>
<programlisting>options KDTRACE_HOOKS
options DDB_CTF</programlisting>
<note>
<para>Users of the AMD64 architecture will want to add the
following line to their kernel configuration file:</para>
<programlisting>options KDTRACE_FRAME</programlisting>
<para>This option provides support for the
<acronym>FBT</acronym> feature. &dtrace; will work without
this option; however, there will be limited support for
function boundary tracing.</para>
</note>
<para>All sources must be rebuilt and installed with
<acronym>CTF</acronym> options.
To accomplish this task, rebuild the &os; sources using:</para>
<!-- XXXTR: WITH_CTF has been reported to leave a user with a
broken system when used with buildworld. Until this is
fixed, comment out those parts. When uncommenting, kill
the extra screen.
-->
<screen>&prompt.root; <userinput>cd /usr/src</userinput>
<!-- &prompt.root; <userinput>make WITH_CTF=1 buildworld</userinput> -->
&prompt.root; <userinput>make WITH_CTF=1 kernel</userinput></screen>
<!-- &prompt.root; <userinput>make WITH_CTF=1 installworld</userinput>
&prompt.root; <userinput>mergemaster -Ui</userinput></screen> -->
<para>The system will need to be restarted.</para>
<para>After rebooting and allowing the new kernel to be loaded
into memory, support for the Korn shell should be added. This
is needed as the &dtrace;Toolkit has several utilities written
in <command>ksh</command>. Install the
<filename role="package">shells/ksh93</filename>. It is also
possible to run these tools under
<filename role="package">shells/pdksh</filename> or
<filename role="package">shells/mksh</filename>.</para>
<para>Finally, obtain the current &dtrace;Toolkit.
If you are running FreeBSD 10, you will find the &dtrace;Toolkit
in <filename>/usr/share/dtrace</filename>.
Otherwise, you can install the &dtrace;Toolkit using the
<filename role="package">sysutils/DTraceToolkit</filename>
port.</para>
</sect1>
<sect1 id="dtrace-using">
<title>Using &dtrace;</title>
<para>Before making use of &dtrace; functionality, the &dtrace;
device must exist. To load the device, issue the following
command:</para>
<screen>&prompt.root; <userinput>kldload dtraceall</userinput></screen>
<para>&dtrace; support should now be available. To view all
probes the administrator may now execute the following
command:</para>
<screen>&prompt.root; <userinput>dtrace -l | more</userinput></screen>
<para>All output is passed to the <command>more</command>
utility as it will quickly overflow the screen buffer. At
this point, &dtrace; should be considered working. It is now
time to review the toolkit.</para>
<para>The toolkit is a collection of ready-made scripts to run
with &dtrace; to collect system information. There are scripts
to check open files, memory, <acronym>CPU</acronym> usage and
a lot more. Extract the scripts with the following
command:</para>
<screen>&prompt.root; <userinput>gunzip -c DTraceToolkit* | tar xvf -</userinput></screen>
<para>Change into that directory with the <command>cd</command>
and change the execution permissions on all files, designated
as those files with lower case names, to
<literal>755</literal>.</para>
<para>All of these scripts will need modifications to their
contents. The ones which refer to
<filename>/usr/bin/ksh</filename> need that changed to
<filename>/usr/local/bin/ksh</filename>, the others which
use <filename>/usr/bin/sh</filename> need to be altered to use
<filename>/bin/sh</filename>, and finally the ones which
use <filename>/usr/bin/perl</filename> will need altered to
use <filename>/usr/local/bin/perl</filename>.</para>
<important>
<para>At this point it is prudent to remind the reader that
&dtrace; support in &os; is <emphasis>incomplete</emphasis>
and <emphasis>experimental</emphasis>. Many of these scripts
will not work as they are either too &solaris;-specific or
use probes which are unsupported at this time.</para>
</important>
<para>At the time of this writing only two of the scripts of the
&dtrace; Toolkit are fully supported in &os;:
the <filename>hotkernel</filename>
and <filename>procsystime</filename> scripts. These are the two
we will explore in the following parts of this section.</para>
<para>The <filename>hotkernel</filename> is designed to identify
which function is using the most kernel time. Run normally, it
will produce output similar to the following:</para>
<screen>&prompt.root; <userinput>cd /usr/share/dtrace/toolkit</userinput>
&prompt.root; <userinput>./hotkernel</userinput>
Sampling... Hit Ctrl-C to end.</screen>
<para>The system administrator must use the
<keycombo action="simul"><keycap>Ctrl</keycap><keycap>C</keycap>
</keycombo> key combination to stop the process. Upon
termination, the script will display a list of kernel functions
and timing information, sorting the output in increasing order
of time:</para>
<screen>kernel`_thread_lock_flags 2 0.0%
0xc1097063 2 0.0%
kernel`sched_userret 2 0.0%
kernel`kern_select 2 0.0%
kernel`generic_copyin 3 0.0%
kernel`_mtx_assert 3 0.0%
kernel`vm_fault 3 0.0%
kernel`sopoll_generic 3 0.0%
kernel`fixup_filename 4 0.0%
kernel`_isitmyx 4 0.0%
kernel`find_instance 4 0.0%
kernel`_mtx_unlock_flags 5 0.0%
kernel`syscall 5 0.0%
kernel`DELAY 5 0.0%
0xc108a253 6 0.0%
kernel`witness_lock 7 0.0%
kernel`read_aux_data_no_wait 7 0.0%
kernel`Xint0x80_syscall 7 0.0%
kernel`witness_checkorder 7 0.0%
kernel`sse2_pagezero 8 0.0%
kernel`strncmp 9 0.0%
kernel`spinlock_exit 10 0.0%
kernel`_mtx_lock_flags 11 0.0%
kernel`witness_unlock 15 0.0%
kernel`sched_idletd 137 0.3%
0xc10981a5 42139 99.3%</screen>
<!-- XXXTR: I attempted to use objdump and nm on /boot/kernel/kernel
to find 0xc10981a5, but to no avail. It would be nice to know
how we should look that up. -->
<para>This script will also work with kernel modules. To use this
feature, run the script with the <option>-m</option>
flag:</para>
<screen>&prompt.root; <userinput>./hotkernel -m</userinput>
Sampling... Hit Ctrl-C to end.
^C
MODULE COUNT PCNT
0xc107882e 1 0.0%
0xc10e6aa4 1 0.0%
0xc1076983 1 0.0%
0xc109708a 1 0.0%
0xc1075a5d 1 0.0%
0xc1077325 1 0.0%
0xc108a245 1 0.0%
0xc107730d 1 0.0%
0xc1097063 2 0.0%
0xc108a253 73 0.0%
kernel 874 0.4%
0xc10981a5 213781 99.6%</screen>
<!-- XXXTR: I was unable to match these up with output from
kldstat and kldstat -v and grep. Maybe I'm missing something
seriously obvious. It is 5AM btw. -->
<para>The <filename>procsystime</filename> script captures and
prints the system call time usage for a given
<acronym>PID</acronym> or process name. In the following
example, a new instance of <filename>/bin/csh</filename>
was spawned. The <filename>procsystime</filename> was executed
and remained waiting while a few commands were typed on the
other incarnation of <command>csh</command>. These are the
results of this test:</para>
<screen>&prompt.root; <userinput>./procsystime -n csh</userinput>
Tracing... Hit Ctrl-C to end...
^C
Elapsed Times for processes csh,
SYSCALL TIME (ns)
getpid 6131
sigreturn 8121
close 19127
fcntl 19959
dup 26955
setpgid 28070
stat 31899
setitimer 40938
wait4 62717
sigaction 67372
sigprocmask 119091
gettimeofday 183710
write 263242
execve 492547
ioctl 770073
vfork 3258923
sigsuspend 6985124
read 3988049784</screen>
<para>As shown, the <function>read()</function> system call
seems to use the most time in nanoseconds with the
<function>getpid()</function> system call used the least amount
of time.</para>
</sect1>
<sect1 id="dtrace-language">
<title>The D Language</title>
<para>The &dtrace; Toolkit includes many scripts in the special
language of &dtrace;. This language is called <quote>the D
language</quote> by &sun; documentation, and it is very similar
to C++. An in depth discussion of the language is beyond the
scope of this document. It is extensively discussed
at <ulink
url="http://wikis.oracle.com/display/DTrace/Documentation"></ulink>.</para>
</sect1>
</chapter>
<!-- XXXTR: Should probably put links and resources here. I'm
nervous about this chapter as it may require a partial
re-write and large modification once DTrace is complete, but
at least we can get everyone started ... -->