464 lines
22 KiB
Text
464 lines
22 KiB
Text
<!--
|
|
The FreeBSD Documentation Project
|
|
-->
|
|
|
|
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook V4.1-Based Extension//EN" [
|
|
<!ENTITY % articles.ent PUBLIC "-//FreeBSD//ENTITIES DocBook FreeBSD Articles Entity Set//EN">
|
|
%articles.ent;
|
|
|
|
<!ENTITY ports "Ports Collection">
|
|
<!ENTITY os.ports "&os; &ports;">
|
|
<!ENTITY frisbee "<application>Frisbee</application>">
|
|
<!ENTITY ghost "<application>Ghost</application>">
|
|
<!ENTITY nessus "<application>Nessus</application>">
|
|
|
|
]>
|
|
|
|
<article>
|
|
<title>Creating a Software Testing Environment Using &os;</title>
|
|
<articleinfo>
|
|
|
|
<authorgroup>
|
|
<author>
|
|
<firstname>Chris</firstname>
|
|
<surname>McMahon</surname>
|
|
<affiliation>
|
|
<address><email>christopher.mcmahon@gmail.com</email></address>
|
|
</affiliation>
|
|
</author>
|
|
</authorgroup>
|
|
|
|
<pubdate>$FreeBSD$</pubdate>
|
|
|
|
<legalnotice id="trademarks" role="trademarks">
|
|
&tm-attrib.freebsd;
|
|
&tm-attrib.cvsup;
|
|
&tm-attrib.intel;
|
|
&tm-attrib.microsoft;
|
|
&tm-attrib.symantec;
|
|
&tm-attrib.xfree86;
|
|
&tm-attrib.general;
|
|
</legalnotice>
|
|
|
|
<abstract><para>A slightly altered version of this paper was first
|
|
published in the Pacific Northwest Software Quality Conference
|
|
Proceedings, 2005.</para></abstract>
|
|
|
|
</articleinfo>
|
|
|
|
|
|
<sect1 id="overview">
|
|
<title>Overview</title>
|
|
|
|
<para>From late 2003 until early 2005, I was a tester in an
|
|
all-&windows; environment. Although unlikely on the face of it,
|
|
&os; became a valuable test tool platform in that context.
|
|
&os; contains useful and powerful applications for any tester
|
|
in any environment.</para>
|
|
|
|
<para>Unlike Linux, &os; is a single monolithic project, rather
|
|
than a collection of disparate parts assembled into a
|
|
distribution. And the most attractive part of &os; for a
|
|
software tester is the &os.ports;—a very large,
|
|
managed set of software applications with a single, simple, and
|
|
uniform installation procedure.</para>
|
|
|
|
<para>This paper describes several software test tools from the
|
|
&os.ports; that I used to test software and
|
|
systems in an all-Windows environment.</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="challenge">
|
|
<title>The Challenge</title>
|
|
|
|
<para>Software testing environments are radically more complex
|
|
than software development environments. Interconnected systems
|
|
to test, network entities, databases, and filesystems present
|
|
challenges to testers that developers can, for the most part, mock
|
|
out and essentially ignore. Software testers need more tools,
|
|
and more complex tools, than do software developers.</para>
|
|
|
|
<para>On the other hand, software development tools are much more
|
|
highly evolved than software testing tools. There is no Eclipse,
|
|
or IntelliJ, or even Visual Studio aimed at testing. Testers
|
|
struggle and scratch to find tools appropriate to their test
|
|
environments and appropriate to their Systems Under Test (SUTs).</para>
|
|
|
|
<para>Every test environment is different. Finding appropriate
|
|
test tools is challenging. And when a tester has identified a
|
|
set of useful tools, introducing them to the test environment
|
|
and integrating them with the test environment is a challenge.</para>
|
|
|
|
</sect1>
|
|
|
|
<sect1 id="freebsd-solution">
|
|
<title>The &os; Solution</title>
|
|
|
|
<sect2 id="freebsd-intro">
|
|
<title>Introduction</title>
|
|
|
|
<para>The set of tools available with the &os; Operating
|
|
System is amazing. The &os; <ulink
|
|
url="http://www.freebsd.org/ports">&ports;</ulink>
|
|
contains more than thirteen thousand separate applications,
|
|
all of which have a standard installation procedure and
|
|
conform to a set of guidelines that make them reliable without
|
|
the need to manage dependencies, appropriate versions, and all
|
|
of the other problems that affect even the most well-managed
|
|
Linux distribution or the various versions of Microsoft
|
|
Windows. The monolithic nature of &os; and the &os.ports;
|
|
removes much of the trouble of integrating
|
|
tools with the test environment, regardless of the OS under
|
|
which the SUT runs. &os; is a highly evolved server
|
|
environment, and contains so many reliable applications, that
|
|
every tester should consider adding a &os; machine (or
|
|
several) to their test environment.</para>
|
|
|
|
<para>Of course, all of the applications available in the
|
|
&os.ports; will not be appropriate for any single
|
|
test environment. Some of the obvious choices for software and
|
|
systems testing are the six hundred or so system utilities,
|
|
the more than one thousand network tools, and the fifty-odd
|
|
benchmarking tools. Whether your test environment is Windows,
|
|
UNIX, Linux, Mac OS, &os; itself, or some combination of
|
|
any of them, &os; and the &os.ports; is a
|
|
great place to look first.</para>
|
|
|
|
</sect2>
|
|
|
|
<sect2 id="freebsd-ports">
|
|
<title>How To Use The Ports System</title>
|
|
|
|
<para>Installing an application from the &os.ports; is a simple matter of: </para>
|
|
|
|
<screen>&prompt.root; <userinput>cd /usr/ports/foo</userinput>
|
|
&prompt.root; <userinput>make install</userinput></screen>
|
|
|
|
<para>and the system does the rest. It reports build status and
|
|
test status, and installs all the relevant documentation as
|
|
well. This aspect of &os; is very attractive to a tester,
|
|
who typically is pressed for time!</para>
|
|
</sect2>
|
|
|
|
<sect2 id="freebsd-testing">
|
|
<title>&os; For Testing</title>
|
|
|
|
<para>The test environment should be more stable than the SUT.
|
|
Once the tester decides to use the tools available on &os;,
|
|
&os;'s long record of reliability makes it an easy choice
|
|
for a test tools platform.</para>
|
|
|
|
<para>My own introduction to &os; occurred when I was hired
|
|
by a major vendor of large-scale network security video
|
|
services to be their network-testing person in an all-Windows
|
|
environment. My first assignment was to replace the obsolete,
|
|
buggy, disk imaging system. I chose to do that with an Open
|
|
Source disk imaging system called <ulink
|
|
url="http://www.cs.utah.edu/flux/papers/frisbee-usenix03-base.html">&frisbee;</ulink>
|
|
which was implemented originally on &os;. I built the
|
|
system—a feature-for-feature replacement for an expensive
|
|
proprietary system—but we never actually used it in our
|
|
production system.</para>
|
|
|
|
<para>In the meantime, I had discovered the &os.ports;
|
|
and started to use some of those tools for testing;
|
|
and I had discovered the power of disk imaging with &frisbee;,
|
|
especially for smoke testing and installation testing; and
|
|
&os; became a permanent part of my test lab. The test lab
|
|
I built, and the &os; systems I created still exist, and
|
|
still provide value to the testers there.</para>
|
|
|
|
</sect2>
|
|
|
|
<sect2 id="freebsd-collab">
|
|
<title>&os; For Collaboration: Twiki</title>
|
|
|
|
<para>A wiki is a simple set of web pages to allow many users to
|
|
share information and collaborate on any sort of documents.
|
|
<ulink url="http://twiki.org">Twiki</ulink> is a wiki fine-tuned for
|
|
document management. It has built-in version control and
|
|
security implemented at the user/password level. I used it
|
|
for requirements management and traceability.</para>
|
|
|
|
<para>The company I worked for managed their requirements in a
|
|
proprietary tool with a truly byzantine hierarchical
|
|
structure. Although the tool was fine for generating
|
|
requirements, it was horrible for testing. Also
|
|
unfortunately, the tool supported very few export mechanisms.
|
|
From the proprietary tool, I had to:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem><simpara>Export to Access</simpara></listitem>
|
|
<listitem><simpara>Export from Access to Excel</simpara></listitem>
|
|
<listitem><simpara>Export from Excel to a delimited file</simpara></listitem>
|
|
<listitem><simpara>Import the delimited file to the wiki by means of a Perl script</simpara></listitem>
|
|
</itemizedlist>
|
|
|
|
<para>And in Twiki I was able to implement version control,
|
|
traceability, and test coverage, all features missing from the
|
|
proprietary tool. This gave me a remarkable ability to trace
|
|
requirements coverage with my tests. Having imported a
|
|
complex hierarchy of sets of requirements, I could use Twiki's
|
|
simple markup features to show each requirement as tested,
|
|
passed, or failed. I could attach test documentation to wiki
|
|
pages, and since I had already scripted the process, it was
|
|
easy to update the requirements as they changed. And Twiki
|
|
itself handled version control for such updates.</para>
|
|
|
|
<para>As with all of the examples in this paper, installing
|
|
Twiki on &os; is fairly simple. It takes just a few
|
|
minutes on a &os; system. However, if you want to use Twiki
|
|
on a Microsoft Windows platform, I strongly suggest you read
|
|
the Twiki documentation extremely carefully. I know someone
|
|
who installed Twiki on Windows, and it took him several days.
|
|
Twiki on Windows requires not only knowledge of Windows, but
|
|
also deep knowledge of Cygwin and Perl.</para>
|
|
|
|
<para>Furthermore, at one point in the project I had to migrate
|
|
my wiki from a machine running &os; 4.8 to one running
|
|
&os; 5.3. The migration consisted merely of installing
|
|
Twiki on &os; 5.3; using <command>tar</command> on the &os; 4.8
|
|
machine to gather all of the Twiki data files specific to my
|
|
testing; FTPing the gathered files to the new &os; 5.3
|
|
machine; and untarring the file. The complete set of Twiki
|
|
documents migrated with no issues or problems at all. That is
|
|
the power of a unified system like &os;.</para>
|
|
</sect2>
|
|
|
|
<sect2 id="freebsd-frisbee">
|
|
<title>&os; For Disk Imaging: Frisbee</title>
|
|
|
|
<para>A disk imaging system is a mechanism for saving and
|
|
restoring all of the data on a physical disk. The most
|
|
popular commercial system for doing this is probably the
|
|
product &ghost;™ from Symantec.</para>
|
|
|
|
<para>The &frisbee; enterprise disk imaging system mentioned above
|
|
had a lot of features I never implemented in the test lab.
|
|
Using &frisbee; and an Open Source tool called <application>PXELINUX</application>, I was
|
|
able to:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem><simpara>Boot the Windows client machines from the network</simpara></listitem>
|
|
<listitem><simpara>Make an image of the client</simpara></listitem>
|
|
<listitem><simpara>Update the BIOS on the client</simpara></listitem>
|
|
<listitem><simpara>Lay down an existing image on the client machine</simpara></listitem>
|
|
<listitem><simpara>Make a set of restore CDs for the client</simpara></listitem>
|
|
</itemizedlist>
|
|
|
|
<para>In the test lab, I only needed to boot from the &frisbee; CD,
|
|
make an image, or lay down an image on the client machine.
|
|
Both &frisbee; and proprietary imaging systems allow the user to
|
|
image individual drives on the client, but I never had a need
|
|
to do this.</para>
|
|
|
|
<para>Installation testing was a large part of my duties at the
|
|
company where I used &os;. To do this testing, I would
|
|
typically use &frisbee; to make an image of a machine containing
|
|
only a Windows OS, install the SUT, and run a smoke test. The
|
|
smoke test typically left the test machine in a very bad
|
|
state. But instead of having to painstakingly clean up the
|
|
mess left by the failed installation, I simply re-imaged the
|
|
machine in question with the bare OS image and started over.
|
|
A typical re-image containing only the Windows OS and a few
|
|
test tools took less than three minutes. Using &frisbee;, we
|
|
could run smoke tests on about six builds per day; before
|
|
&frisbee;, we could run smoke tests on about three builds per
|
|
week.</para>
|
|
|
|
<para>Of course, &ghost; or other proprietary tools also image
|
|
machines quickly under these circumstances: once you buy the
|
|
tool, license the software, install it on an appropriate
|
|
server, and configure it properly. I prefer &frisbee; to &ghost;
|
|
because: &frisbee; is marginally faster; &frisbee; is very easy to
|
|
install on &os;; and &frisbee; is very efficient. Adding a
|
|
couple of small Perl scripts to the normal &frisbee;
|
|
distribution gave me an imaging environment tailored for the
|
|
test lab.</para>
|
|
|
|
<para>I also used &frisbee; to preserve the state of a machine
|
|
after I had uncovered particularly complex defects. That is,
|
|
if it took a large effort (many steps and/or a long duration
|
|
of time) to demonstrate a defect, I could make an image of the
|
|
machine at the point at which the defect was visible, and
|
|
restore the image at will to demonstrate the defect to
|
|
developers or managers.</para>
|
|
</sect2>
|
|
|
|
<sect2 id="freebsd-nessus">
|
|
<title>&os; Security Testing: &nessus;</title>
|
|
|
|
<para>Whenever you have more than one entity on a network, and
|
|
whenever you expose a server to the wider Internet, security
|
|
of the machine itself is always a concern. <ulink
|
|
url="http://www.nessus.org">&nessus;</ulink> is an Open Source
|
|
remote vulnerability scanner for security and penetration
|
|
testing that consistently is rated among the top products of
|
|
its type throughout the security industry.</para>
|
|
|
|
<para>&nessus; probes a remote machine over the network for
|
|
security vulnerabilities. It does a port scan, finds which
|
|
ports are open, and investigates the software that has those
|
|
ports open for a huge number of security risks, for all major
|
|
OSs. It generates detailed reports in a number of formats
|
|
that anyone can understand. The number of security probes
|
|
available in the default installation of &nessus; is very large,
|
|
but sophisticated security and penetration testers take
|
|
advantage of NASL, the &nessus; Attack Scripting Language, to
|
|
craft their own attacks using &nessus;' available features.</para>
|
|
|
|
<para>Of interest is that, while &nessus; is a free download for
|
|
UNIX-like systems (and is available in the &ports; of
|
|
&os;), it is available on Windows only as a commercial
|
|
product from a company called <emphasis>Tenable</emphasis>. The Tenable product is
|
|
<application>NeWT</application>, <quote>Nessus on Windows Technology</quote>.</para>
|
|
</sect2>
|
|
|
|
<sect2 id="freebsd-network">
|
|
<title>&os; Network Tools</title>
|
|
|
|
<para>&os; is most widely used as a robust server platform.
|
|
It follows, then, that tools related to network analysis and
|
|
performance will be highly evolved on &os;. Here is a
|
|
brief description of network diagnostic tools that I found
|
|
invaluable in testing in a networked environment.</para>
|
|
|
|
<para>From the name, one would assume that <ulink
|
|
url="http://www.ntop.org">ntop</ulink> emulates the functions of
|
|
the UNIX &man.top.1; command, but for the network
|
|
rather than for the local machine. Perhaps the first version
|
|
did; currently, <application>ntop</application> is capable of providing detailed
|
|
information about a huge number of hosts and their status and
|
|
activities on the network.</para>
|
|
|
|
<para>For testing, two features I found very powerful: at a high
|
|
level, <application>ntop</application> shows the amount of network
|
|
traffic on the entire network segment minute-by-minute,
|
|
hour-by-hour, and day-by-day in a graphical format. Also,
|
|
<application>ntop</application> shows information about recent connections between
|
|
individual hosts on the network.</para>
|
|
|
|
<para>It is easy to see traffic trends on the network as they
|
|
are occurring; also, if something anomalous appears, <application>ntop</application>
|
|
records detailed information about network connections between
|
|
hosts, including the ports over which the connection happened.
|
|
This was critically important when analyzing software
|
|
issues. If <application>ntop</application> showed a period of time for which traffic was
|
|
particularly high, I would find out which host was generating
|
|
the traffic. I would examine the software running on that
|
|
host, over that port. Often it was a new build with a
|
|
bug.</para>
|
|
|
|
<para><ulink url="http://ettercap.sourceforge.net">Ettercap</ulink> is
|
|
a tool for ARP poisoning which can also decipher passwords on
|
|
the fly and corrupt IP traffic by means of a Man In The Middle
|
|
(MITM) attack. However, I used <application>Ettercap</application> as a performance tool.
|
|
In my test labs, all of my &os; machines ran on discarded
|
|
hardware, Pentium II processors. I found that when I used
|
|
<application>Ettercap</application> to sniff traffic between two hosts, the lack of
|
|
processing power caused <application>Ettercap</application> on the slow MITM machine to
|
|
start dropping packets, making it look to the client machine
|
|
in the SUT as if there was interference or other trouble on
|
|
the network. And by varying the load on the &os; machine,
|
|
I could in fact control the number of packets being dropped:
|
|
running <application>Ettercap</application> and the UNIX <command>yes</command> utility
|
|
caused 100% packet loss.</para>
|
|
|
|
<para>This was my most creative use of a &os; tool for
|
|
testing. In a more straightforward application, any time a
|
|
tester needs to eavesdrop on traffic between two hosts on a
|
|
network, <application>Ettercap</application> is an excellent choice because of its power
|
|
and ease of use.</para>
|
|
|
|
<para>Perl gets a special mention here because Perl's network
|
|
utilities are outstandingly good compared to other
|
|
languages. Perl <literal>Net::*</literal> modules and
|
|
<literal>IO::Socket::*</literal> modules are robust and
|
|
powerful—but they often fail to compile on Windows. It is
|
|
the ease of use of Perl's network utilities on &os; that
|
|
gets Perl the mention in this section.</para>
|
|
|
|
<para>I use Perl's network utilities to impersonate network
|
|
clients and servers for test purposes. On one occasion, I was
|
|
required to test software that was a client to an interface on
|
|
the <emphasis>New York Stock Exchange</emphasis>. Unfortunately, the NYSE test
|
|
server was down about nine days out of ten. I wrote a little
|
|
network server in Perl to emulate simple functions of the NYSE
|
|
server in order to test the client software.</para>
|
|
|
|
<para>On another occasion, I had to test some functions on a web
|
|
server. It was not difficult to write a simple Perl HTTP
|
|
client to validate that the server was functioning properly.</para>
|
|
|
|
<para>I have also used Perl to validate the output from a server
|
|
sending to a multicast address. I wrote a simple Perl
|
|
multicast client on &os; to monitor the traffic on a
|
|
multicast address. Lincoln Stein's excellent
|
|
<literal>IO::Socket::Multicast</literal> module made it easy.
|
|
(Note: I never got <literal>IO::Socket::Multicast</literal> to
|
|
compile on Windows. I tried it on Windows 2000 and on Windows
|
|
XP.)</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<sect1 id="conclusion">
|
|
<title>Conclusion</title>
|
|
|
|
<para>I used tools from the &os.ports; in four
|
|
areas: in the network, where the operating system has very
|
|
little impact on how software behaves; for remote security
|
|
testing and performance testing in order to manipulate remote
|
|
machines over the network, regardless of the operating system;
|
|
for disk imaging of Windows, Linux, and &os; machines; and on
|
|
the webserver, where Twiki was my collaboration tool of choice.</para>
|
|
|
|
<para>Because the installation procedure for all of these tools is
|
|
standard, I spent relatively little time installing and
|
|
configuring my tools. Because all the tools were hosted on a
|
|
single platform, I had all of my configuration and diagnostic
|
|
information located in just a few places. I kept all of my
|
|
potentially dangerous security tools on a single machine, which
|
|
made my presence on the network tolerable to the company's
|
|
network management staff. And the compatibility between &os;
|
|
versions made it fairly simple to upgrade and to manage multiple
|
|
&os; machines. And of course, I could rely on the
|
|
correctness of my test results, because the system itself is so
|
|
reliable.</para>
|
|
|
|
<para>I have tried using Linux in a similar way, but my experience
|
|
is that package management quickly becomes tedious if not
|
|
overwhelming. The &os.ports; handled that for me.
|
|
And many of these tools are simply not available on Microsoft
|
|
Windows. And when they (or their equivalents) are available,
|
|
their cost, both financial and in terms of overhead, was simply
|
|
too high.</para>
|
|
|
|
<para>&os;'s simple installation procedures and robust &ports;
|
|
makes it easy to experiment with the huge number of
|
|
tools available. I often find myself browsing the &ports;
|
|
looking for interesting applications to install, just
|
|
to see how they work. (I found <application>Ettercap</application> by browsing the &ports;,
|
|
a tool that became very useful very quickly.) It
|
|
became clear that the more tools I used on &os;, the more
|
|
economical became the management of those tools.</para>
|
|
|
|
<para>The next time you need to reach into your toolbox for some
|
|
sophisticated, reliable, and powerful testing tools, I hope you
|
|
find them in &os;.</para>
|
|
</sect1>
|
|
|
|
<!--
|
|
<sect1>
|
|
<title>Biography</title>
|
|
|
|
<para>Chris McMahon has been for nearly a decade a tester of library
|
|
software, telecom, stock trading, and networked video systems. He
|
|
has published a number of articles aimed at technical testers, but
|
|
his true career is at being an enthusiastic beginner. Chris works
|
|
for ThoughtWorks, Inc. as a tester, and can be reached at
|
|
christopher.mcmahon@gmail.com.</para>
|
|
</sect1>
|
|
-->
|
|
|
|
</article>
|