<!--
     The FreeBSD Documentation Project
-->

<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook V4.1-Based Extension//EN" [
<!ENTITY branches.ascii SYSTEM "branches.ascii">
<!ENTITY % authors PUBLIC "-//FreeBSD//ENTITIES DocBook Author Entities//EN">
%authors;
<!ENTITY % teams PUBLIC "-//FreeBSD//ENTITIES DocBook Team Entities//EN">
%teams;
<!ENTITY % mailing-lists PUBLIC "-//FreeBSD//ENTITIES DocBook Mailing List Entities//EN">
%mailing-lists;
<!ENTITY % man PUBLIC "-//FreeBSD//ENTITIES DocBook Manual Page Entities//EN">
%man;
<!ENTITY % freebsd PUBLIC "-//FreeBSD//ENTITIES DocBook Miscellaneous FreeBSD Entities//EN">
%freebsd;
<!ENTITY art.re.pkgs '<ulink url="../releng-packages/article.html">The Release Engineering of Third Party Packages</ulink>'>
]>

<article>
  <title>FreeBSD Release Engineering</title>
  <articleinfo>

    <!-- This paper was presented at BSDCon Europe in Brighton, UK on
         November 11, 2001 -->
    <confgroup>
      <confdates>November 2001</confdates>
      <conftitle>BSDCon Europe</conftitle>
    </confgroup>

    <authorgroup>
      <author>
        <firstname>Murray</firstname>
        <surname>Stokely</surname>
        <authorblurb>
          <para>I've been involved in the development of FreeBSD based products
          since 1997 at Walnut Creek CDROM, BSDi, and now Wind River Systems.
          FreeBSD 4.4 was the first official release of FreeBSD that I played
          a significant part in.</para>
        </authorblurb>
        <affiliation>
          <address><email>murray@FreeBSD.org</email>
          <otheraddr><ulink
          url="http://www.FreeBSD.org/~murray">http://www.FreeBSD.org/~murray</ulink></otheraddr>
          </address>
        </affiliation>
      </author>
    </authorgroup>

    <pubdate>$FreeBSD$</pubdate>
    <abstract>
      <para>This paper describes the approach used by the FreeBSD
        release engineering team to make production quality releases
        of the FreeBSD Operating System.  It details the methodology
        used for the release of FreeBSD 4.4 and describes the tools
        available for those interested in producing customized FreeBSD
        releases for corporate rollouts or commercial
        productization.</para>
    </abstract>

  </articleinfo>
  
<!-- Introduction -->
<sect1 id="introduction">
  <title>Introduction</title>

  <para>The development of FreeBSD is a very open process.  FreeBSD is
    comprised of contributions from thousands of people around the
    world.  The FreeBSD Project provides anonymous
    <acronym>CVS</acronym>[1] access to the general public so that
    others can have access to log messages, diffs (patches) between
    development branches, and other productivity enhancements that
    formal source code management provides.  This has been a huge help
    in attracting more talented developers to FreeBSD.  However, I
    think everyone would agree that chaos would soon manifest if write
    access was opened up to everyone on the Internet.  Therefore only
    a <quote>select</quote> group of nearly 300 people are given write
    access to the <acronym>CVS</acronym> repository.  These
    <emphasis>committers[6]</emphasis> are responsible for the bulk of
    FreeBSD development.  An elected <emphasis>core-team[7]</emphasis>
    of very senior developers provides some level of direction over
    the project.</para>
  
  <para>The rapid pace of <systemitem
    class="osname">FreeBSD</systemitem> development leaves little time
    for polishing the development system into a production quality
    release.  To solve this dilemma, development continues on two
    parallel tracks.  The main development branch is the
    <emphasis>HEAD</emphasis> or <emphasis>trunk</emphasis> of our CVS
    tree, known as <quote>FreeBSD-CURRENT</quote> or
    <quote>-CURRENT</quote> for short.</para>
  
  <para>A more stable branch is maintained, known as
    <quote>FreeBSD-STABLE</quote> or <quote>-STABLE</quote> for short.
    Both branches live in a master CVS repository in California and
    are replicated via <application
    class="software">CVSup</application>[2] to mirrors all over the
    world.  FreeBSD-CURRENT[8] is the <quote>bleeding-edge</quote> of
    FreeBSD development where all new changes first enter the system.
    FreeBSD-STABLE is the development branch from which major releases
    are made.  Changes go into this branch at a different pace, and
    with general assumption that they have first gone into
    FreeBSD-CURRENT and have been thoroughly tested by our user
    community.</para>
  
  <para>In the interim period between releases, nightly snapshots are
    built automatically by the FreeBSD Project build machines and made
    available for download from <systemitem
    class="resource">ftp://stable.FreeBSD.org/</systemitem>.  The
    widespread availability of binary release snapshots, and the
    tendency of our user community to keep up with -STABLE development
    with CVSup and <quote><command>make</command>
    <MakeTarget>world</MakeTarget></quote>[8] helps to keep
    FreeBSD-STABLE in a very reliable condition even before the
    quality assurance activities ramp up pending a major
    release.</para>
  
  <para>Bug reports and feature requests are continuously submitted by
    users throughout the release cycle.  Problems reports are entered into our
    <application class="software">GNATS</application>[9] database
    through email, the &man.send-pr.1; application, or via the web
    interface provided at <ulink
    url="http://www.FreeBSD.org/send-pr.html"></ulink>
    In addition to the multitude of different technical mailing lists
    about FreeBSD, the &a.qa; provides a forum for discussing the finer
    points of <quote>release-polishing</quote>.</para>
  
  <para>To service our most conservative users, individual release
    branches were introduced with FreeBSD 4.3.
    These release branches are created shortly before a final release
    is made.  After the release goes out, only the most critical
    security fixes and additions are merged onto the release branch.
    In addition to source updates via CVS, binary patchkits are
    available to keep systems on the <emphasis>RELENG_4_3 </emphasis>
    and <emphasis>RELENG_4_4</emphasis> branches updated.</para>
  
  <para><xref linkend="release-proc"> discusses the
    different phases of the release engineering process leading up to
    the actual system build and <xref linkend="release-build">
    describes the actual build process.  <xref
    linkend="extensibility"> describes how the base
    release may be extended by third parties and <xref
    linkend="lessons-learned"> details some of the
    lessons learned through the release of FreeBSD 4.4.  Finally,
    <xref linkend="future"> presents future directions
    of development.</para>
</sect1>

<!-- Release Process -->
<sect1 id="release-proc">
  <title>Release Process</title>

  <para>New releases of FreeBSD are released from the -STABLE branch
    at approximately four month intervals.  The FreeBSD release
    process begins to ramp up 45 days before the anticipated release
    date when the release engineer sends an email to the development
    mailing lists to remind developers that they only have 15 days to
    integrate new changes before the code freeze.  During this time,
    many developers perform what have become know as <quote>MFC
    sweeps</quote>.  <acronym>MFC</acronym> stands for <quote>Merge
    From CURRENT</quote> and it describes the process of merging a
    tested change from our -CURRENT development branch to our -STABLE
    branch.</para>
  
  <sect2>
    <title>Code Review</title>

    <para>Thirty days before the anticipated release, the source
      repository enters a <quote>code slush</quote>.  During this
      time, all commits to the -STABLE branch must be approved by the
      &a.re;.  The kinds of changes that are allowed during this 15 day
      period include:</para>
    
    <itemizedlist>
      <listitem>
        <para>Bug fixes.</para>
      </listitem>
      
      <listitem>
        <para>Documentation updates.</para>
      </listitem>
      
      <listitem>
        <para>Security-related fixes of any kind.</para>
      </listitem>
      
      <listitem>
        <para>Minor changes to device drivers, such as adding new Device
        IDs.</para>
      </listitem>
      
      <listitem>
        <para>Any additional change that the release engineering team feels
        is justified, given the potential risk.</para>
      </listitem>
    </itemizedlist>
    
    <para>After the first 15 days of the code slush, a
      <emphasis>release candidate</emphasis> is released for
      widespread testing and the code enters a <quote>code
      freeze</quote> where it becomes much harder to justify new
      changes to the system unless a serious bug-fix or security issue
      is involved.  During the code freeze, at least one release
      candidate is released per week, until the final release is
      ready.  During the days leading to the final release, the
      release engineering team is in constant communication with the
      security-officer team, the documentation maintainers, and the
      port maintainers, to ensure that all of the
      different components required for a successful release are
      available.</para>
  </sect2>
  
  <sect2>
    <title>Final Release Checklist</title>

    <para>When several release candidates have been made available for
      widespread testing and all major issues have been resolved, the
      final release <quote>polishing</quote> can begin.</para>
    
    <sect3>
      <title>Creating the Release Branch</title>

      <para>As described in the introduction, the <literal>RELENG_X_Y</literal> release
        branch is a relatively new addition to our release engineering
        methodology.  The first step in creating this branch is to
        ensure that you are working with the newest version of the
        <literal>RELENG_X</literal> sources that you want to branch
        <emphasis>from</emphasis>.</para>
      
      <screen>/usr/src&prompt.root; <userinput>cvs update -rRELENG_4 -P -d</userinput></screen>
      
      <para>The next step is to create a branch point
        <emphasis>tag</emphasis>, so that diffs against the start of
        the branch are easier with CVS:</para>
      
      <screen>/usr/src&prompt.root; <userinput>cvs rtag -rRELENG_4 RELENG_4_4_BP src</userinput></screen>
      
      <para>And then a new branch tag is created with:</para>
      
      <screen>/usr/src&prompt.root; <userinput>cvs rtag -b -rRELENG_4_4_BP RELENG_4_4 src</userinput></screen>
      
      <note>
        <para><emphasis>The <literal>RELENG_*</literal> tags are
          restricted for use by the CVS-meisters and release
          engineers.</emphasis></para>
      </note>
      
      <sidebar>
        <para>A <quote><emphasis>tag</emphasis></quote> is CVS
        vernacular for a label that identifies the source at a specific point
        in time.  By tagging the tree, we ensure that future release builders
        will always be able to use the same source we used to create the 
        official FreeBSD Project releases.</para>
      </sidebar>
      
      <mediaobject>
        <imageobject>
          <imagedata fileref="branches" align="center"> 
        </imageobject>

        <textobject>
          <literallayout>
          &branches.ascii;
          </literallayout>
        </textobject>
    
        <textobject>
          <phrase>FreeBSD Development Branches</phrase> 
        </textobject>
      </mediaobject>

    </sect3>
    
    <sect3 id="versionbump">
      <title>Bumping up the Version Number</title>

      <para>Before the final release can be tagged, built, and
        released, the following files need to be modified to reflect
        the correct version of FreeBSD:</para>

      <itemizedlist>
        <listitem>
          <para><filename>doc/en_US.ISO8859-1/books/handbook/mirrors/chapter.sgml
          </filename></para>
        </listitem>

        <listitem>
          <para><filename>doc/share/sgml/freebsd.ent</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/Makefile.inc</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/UPDATING</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/gnu/usr.bin/groff/tmac/mdoc.local</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/release/Makefile</filename></para>
        </listitem>
      
        <listitem>
          <para><filename>src/release/doc/en_US.ISO8859-1/share/sgml/release.dsl</filename></para>
        </listitem>
      
        <listitem>
          <para><filename>src/release/doc/share/examples/Makefile.relnotesng</filename></para>
        </listitem>
      
        <listitem>
          <para><filename>src/release/doc/share/sgml/release.ent</filename></para>
        </listitem>
      
        <listitem>
          <para><filename>src/sys/conf/newvers.sh</filename></para>
        </listitem>
      
        <listitem>
          <para><filename>src/sys/sys/param.h</filename></para>
        </listitem>
      
        <listitem>
          <para><filename>www/en/releases/*</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/en/docs.sgml</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/en/cgi/ports.cgi</filename></para>
        </listitem>

      </itemizedlist>
    
      <para>The release notes and errata files also need to be adjusted for the
      new release (on the release branch) and truncated appropriately 
      (on the stable/current branch):</para>
    
      <itemizedlist>
        <listitem>
          <para><filename>src/release/doc/en_US.ISO8859-1/relnotes/common/new.sgml
          </filename></para>
        </listitem>
      
        <listitem>
          <para><filename>src/release/doc/en_US.ISO8859-1/errata/article.sgml
          </filename></para>
        </listitem>
      </itemizedlist>

      <para><application>Sysinstall</application> should be updated to note 
        the number of available ports and the amount of disk space required
	for the Ports Collection.  This information is currently kept in 
	<filename>src/release/sysinstall/dist.c</filename>.</para>
    </sect3>
    
    <sect3>
      <title>Creating Release Tags</title>

      <para>When the final release is ready, the following command
        will create the <literal>RELENG_4_4_0_RELEASE</literal>
        tag.</para>
      
      <screen>
      /usr/src&prompt.root; <userinput>cvs rtag -rRELENG_4_4 RELENG_4_4_0_RELEASE src</userinput>
      </screen>
      
      <para>The Documentation and Ports managers are responsible for
        tagging the respective trees with the <literal>RELEASE_4_4_0</literal>
        tag.</para>
      
      <para>Occasionally, a last minute fix may be required
        <emphasis>after</emphasis> the final tags have been created.
        In practice this isn't a problem, since <acronym>CVS</acronym>
        allows tags to be manipulated with <command>cvs
        tag -d <replaceable>tagname filename</replaceable></command>.
        It is very important that any last minute changes be tagged
        appropriately as part of the release.  FreeBSD releases must
        always be reproduceable.  Local hacks in the release
        engineer's environment are not acceptable.</para>
    </sect3>
  </sect2>
</sect1>

<!-- Release Building -->
<sect1 id="release-build">
  <title>Release Building</title>

  <para>FreeBSD <quote>releases</quote> can be built by anyone with a
    fast machine and access to a source repository. (That should be
    everyone, since we offer anonymous CVS! See The Handbook for
    details.).  The <emphasis>only</emphasis> special requirement is
    that the <devicename>vn</devicename> (<emphasis>On -CURRENT, this
    device has been replaced by the new <devicename>md</devicename>
    memory disk driver </emphasis>.)  device must be available. If the
    device is not loaded into your kernel, then the kernel module
    should be automatically loaded when &man.vnconfig.8; is executed
    during the boot media creation phase.  All of the tools necessary
    to build a release are available from the CVS repository in
    <filename>src/release</filename>.  These tools aim to provide a
    consistent way to build FreeBSD releases.  A complete release can
    actually be built with only a single command, including the
    creation of <acronym>ISO</acronym> images suitable for burning to
    CDROM, installation floppies, and an FTP install directory.  This
    command is aptly named <quote><command>make
    release</command></quote>.</para>
  
  <sect2>
    <title><quote>make release</quote></title>

    <para>To successfully build a release, you must first populate
      <filename>/usr/obj</filename> by running <quote><command>make
      world</command></quote> or simply
      <quote><command>make
      buildworld</command></quote>. The release
      target requires several variables be set properly to build a
      release:</para>
    
    <itemizedlist>
      <listitem>
        <para><makevar>CHROOTDIR</makevar> - The directory to be used as the
        chroot environment for the entire release build.</para>
      </listitem>
      
      <listitem>
        <para><makevar>BUILDNAME</makevar> - The name of the release to be
        built.</para>
      </listitem>
      
      <listitem>
        <para><makevar>CVSROOT</makevar> - The location of a CVS Repository.
        </para>
      </listitem>
      
      <listitem>
        <para><makevar>RELEASETAG</makevar> - The CVS tag corresponding to the
        release you would like to build.</para>
      </listitem>    
    </itemizedlist>
    
     <para>If you do not already have access to a local CVS
       repository, then you may mirror one with <ulink
       url="http://www.FreeBSD.org/handbook/synching.html#CVSUP">CVSup</ulink>.
       The supplied supfile,
       <filename>/usr/share/examples/cvsup/cvs-supfile</filename>, is
       a useful starting point for mirroring the CVS
       repository.</para>

     <para>If <makevar>RELEASETAG</makevar> is omitted, then the
       release will be built from the HEAD (a.k.a. -CURRENT) branch.
       Releases built from this branch are normally referred to as
       <quote>-CURRENT snapshots</quote>.</para>

     <para>There are many other variables available to customize the
       release build. Most of these variables are documented at the
       top of <filename>src/release/Makefile</filename>. The exact
       command used to build the official FreeBSD 4.4 (x86) release
       was:</para>
     
     <screen><command>make <literal>release CHROOTDIR=/local3/release \
       BUILDNAME=4.4-RELEASE \
       CVSROOT=/host/cvs/usr/home/ncvs \
       RELEASETAG=RELENG_4_4_0_RELEASE</literal>
       </command>
     </screen>

     <para>The release Makefile can be broken down into several distinct
       steps.</para>
     
    <itemizedlist>
      <listitem>
        <para>Creation of a sanitized system environment in a separate 
	directory hierarchy with <quote><command>make 
	<literal>installworld</literal></command></quote>.
        </para>
      </listitem>
      
      <listitem>
        <para>Checkout from CVS of a clean version of the system source, 
        documentation, and ports into the release build hierarchy.</para>
      </listitem>
      
      <listitem>
        <para>Population of <filename>/etc</filename> and 
        <filename>/dev</filename> in the chrooted
        environment.</para>
      </listitem>
      
      <listitem>
        <para>chroot into the release build hierarchy, to make it harder for 
        the outside environment to taint this build.</para>
      </listitem>
      
      <listitem>
        <para><quote><command>make world</command></quote> 
        in the chrooted environment.</para>
      </listitem>
      
      <listitem>
        <para>Build of Kerberos-related binaries.</para>
      </listitem>
      
      <listitem>
        <para>Build <quote><filename>GENERIC</filename></quote> kernel.</para>
      </listitem>
      
      <listitem>
        <para>Creation of a staging directory tree where the binary 
        distributions will be built and packaged.</para>
      </listitem>
      
      <listitem>
        <para>Build and installation of the documentation toolchain needed to 
        convert the documentation source (SGML) into HTML and text documents 
        that will accompany the release.</para>
      </listitem>
      
      <listitem>
        <para>Build and installation of the actual documentation 
        (user manuals, tutorials, release notes, hardware compatibility lists, 
        and so on.)</para>
      </listitem>
      
      <listitem>
        <para>Build of the <quote>crunched</quote> binaries used for 
        installation floppies.</para>
      </listitem>
      
      <listitem>
        <para>Package up distribution tarballs of the binaries and sources.
        </para>
      </listitem>
      
      <listitem>
        <para>Create the boot media and a <quote>fixit</quote> floppy.</para>
      </listitem>
      
      <listitem>
        <para>Create FTP installation hierarchy.</para>
      </listitem>
      
      <listitem>
        <para><emphasis>(optionally)</emphasis> Create ISO images for 
        CDROM/DVD media.</para>
      </listitem>
    </itemizedlist>

    <para>For more information about the release build infrastructure,
      please see &man.release.7;.</para>

  </sect2>
  
  <sect2>
    <title>Building XFree86</title>

    <para>XFree86 is an important component for many desktop users.
      The easiest way to build XFree86 is to use the
      <filename>src/release/scripts/X11/build_x.sh</filename> script.
      This script requires that XFree86 and Tcl/Tk already be
      installed on the build host.  After compiling the necessary X
      servers, the script will package all of the files into tarballs
      that &man.sysinstall.8; expects to find in the
      <filename>XF86336</filename> directory of the installation
      media.</para>

    <note><para>It is important to remove any site-specific settings
      from <filename>/etc/make.conf</filename>.  For example, it would
      be unwise to distribute binaries that were built on a system
      with <varname>CPUTYPE</varname> set to a specific
      processor.</para></note>
    
  </sect2>

  <sect2>
    <title>Contributed Software (<quote>ports</quote>)</title>

    <para>The <ulink url="http://www.FreeBSD.org/ports">FreeBSD Ports
      collection</ulink> is a collection of over &os.numports;
      third-party software packages available for FreeBSD. The &a.portmgr;
      is responsible for maintaining a consistent ports tree that can be used
      to create the binary packages that accompany official FreeBSD
      releases.</para>

    <para>The release engineering activities for our collection of
      third-party packages is beyond the scope of this document.  A
      separate article, &art.re.pkgs;, covers this topic
      in depth.</para>

  </sect2>
  
  <sect2>
    <title>Release ISOs</title>

    <para>Starting with FreeBSD 4.4, the FreeBSD Project decided to
      release all four ISO images that were previously sold on the
      <emphasis>BSDi/Wind River Systems/FreeBSD Mall</emphasis>
      <quote>official</quote> CDROM distributions. Each of the four
      discs must contain a <filename>README.TXT</filename> file that
      explains the contents of the disc, a
      <filename>CDROM.INF</filename> file that provides meta-data for
      the disc so that &man.sysinstall.8; can validate and use the
      contents, and a <filename>filename.txt</filename> file that
      provides a manifest for the disc. This
      <emphasis>manifest</emphasis> can be created with a simple
      command:</para>
    
    <screen>/stage/cdrom&prompt.root; <userinput>find . -type f | sed -e 's/^\.\///' | sort > filename.txt</userinput></screen>
    
    <para>The specific requirements of each CD are outlined below.</para>
    
    <sect3>
      <title>Disc 1</title>

      <para>The first disc is almost completely created by
        <quote><command>make
        <literal>release</literal></command></quote>. The only changes
        that should be made to the disc1 directory are the addition of
        a <filename>tools</filename> directory, <application
        class="software">XFree86</application>, and as many popular
        third party software packages as will fit on the disc. The
        <filename>tools</filename> directory contains software that allow users to create
        installation floppies from other operating systems. This disc
        should be made bootable so that users of modern PCs do not
        need to create installation floppy disks.</para>
      
      <para>If an alternate version of XFree86 is to be provided, then
        &man.sysinstall.8; must be updated to reflect the new location
        and installation instructions. The relevant code is contained
        in <filename>src/release/sysinstall</filename> on -STABLE or
        <filename>src/usr.sbin/sysinstall</filename> on
        -CURRENT. Specifically, the files <filename>dist.c</filename>,
        <filename>menus.c</filename>, and
        <filename>config.c</filename> will need to be updated.</para>
    </sect3>
    
    <sect3>
      <title>Disc 2</title>

      <para>The second disc is also largely created by <quote>make
        release</quote>. This disc contains a <quote>live
        filesystem</quote> that can be used from &man.sysinstall.8; to
        troubleshoot a FreeBSD installation. This disc should be
        bootable and should also contain a compressed copy of the CVS
        repository in the <filename>CVSROOT</filename> directory and
        commercial software demos in the <filename>commerce</filename>
        directory.</para>
    </sect3>
    
    <sect3>
      <title>Discs 3 and 4</title>

      <para>The remaining two discs contain additional software
        packages for FreeBSD. The packages should be clustered so that
        a package and all of its <emphasis>dependencies</emphasis> are
        included on the same disc.  More information about the
        creation of these discs is provided in the &art.re.pkgs;
        article.</para>
    </sect3>
  </sect2>
</sect1>

<!-- Extensibility -->
<sect1 id="extensibility">
  <title>Extensibility</title>

  <para>Although FreeBSD forms a complete operating system, there is
    nothing that forces you to use the system exactly as we have
    packaged it up for distribution. We have tried to design the
    system to be as extensible as possible so that it can serve as a
    platform that other commercial products can be built on top
    of. The only <quote>rule</quote> we have about this is that if you
    are going to distribute FreeBSD with non-trivial changes, we
    encourage you to document your enhancements! The FreeBSD community
    can only help support users of the software we provide. We
    certainly encourage innovation in the form of advanced
    installation and administration tools, for example, but we can't
    be expected to answer questions about it.</para>
  
  <sect2>
    <title>Creating Customized Boot floppies</title>

    <para>Many sites have complex requirements that may require
      additional kernel modules or userland tools be added to the
      installation floppies.  The <quote>quick and dirty</quote> way
      to accomplish this would be to modify the staging directory of
      an existing <quote>make release</quote> build hierarchy:</para>
    
    <itemizedlist>
      <listitem>
        <para>Apply patches or add additional files inside the chroot
          release build directory.</para>
      </listitem>
      
      <listitem>
        <para><command>rm 
        ${CHROOTDIR}/usr/obj/usr/src/release/release.[48]</command></para>
      </listitem>
      
      <listitem>
        <para>rebuild &man.sysinstall.8;, the kernel, or whatever
          parts of the system your change affected.</para>
      </listitem>
      
      <listitem>
        <para><command>chroot ${CHROOTDIR} ./mk release.4
        </command></para>
      </listitem>
      
      <listitem>
        <para><command>chroot ${CHROOTDIR} ./mk release.8
        </command></para>
      </listitem>
    </itemizedlist>
    
    <para>New release floppies will be located in
      <filename>${CHROOTDIR}/R/stage/floppies</filename>.</para>
    
    <para>Alternatively, the
      <quote><filename>boot.flp</filename></quote> make
      target can be called, or the filesystem
      creating script,
      <filename>src/release/scripts/doFS.sh</filename>, may be invoked
      directly.</para>
    
    <para>Local patches may also be supplied to the release build by
      defining the <makevar>LOCAL_PATCH</makevar> variable in <quote>make
      release</quote>.
    </para>
  </sect2>
  
  <sect2>
    <title>Scripting <command>sysinstall</command></title>

    <para>The FreeBSD system installation and configuration tool,
      &man.sysinstall.8;, can be scripted to provide automated installs
      for large sites. This functionality can be used in conjunction
      with Intel's PXE[13] to bootstrap systems from the network, or
      via custom boot floppies with a sysinstall script.  An example
      sysinstall script is available in the CVS tree as
      <filename>src/release/sysinstall/install.cfg</filename>.</para>
  </sect2>
</sect1>

<!-- Lessons Learned -->
<sect1 id="lessons-learned">
  <title>Lessons Learned from FreeBSD 4.4</title>

  <para>The release engineering process for 4.4 formally began on
    August 1st, 2001. After that date all commits to the
    <literal>RELENG_4</literal> branch of FreeBSD had to be explicitly
    approved by the &a.re;.  The first
    release candidate for the x86 architecture was released on August
    16, followed by 4 more release candidates leading up to the final
    release on September 18th.  The security officer was very involved
    in the last week of the process as several security issues were
    found in the earlier release candidates. A total of over
    <emphasis>500</emphasis> emails were sent to the &a.re; in
    little over a month.</para>
  
  <para>Our user community has made it very clear that the security
    and stability of a FreeBSD release should not be sacrificed for
    any self-imposed deadlines or target release dates.  The FreeBSD
    Project has grown tremendously over its lifetime and the need for
    standardized release engineering procedures has never been more
    apparent. This will become even more important as FreeBSD is
    ported to new platforms.</para>
</sect1>

<!-- Future Directions -->
<sect1 id="future">
  <title>Future Directions</title>

  <para>It is imperative for our release engineering activities to
    scale with our growing userbase. Along these lines we are working
    very hard to document the procedures involved in producing FreeBSD
    releases.</para>
  
  <itemizedlist>
    <listitem>
      <para><emphasis>Parallelism</emphasis> - Certain portions of the
        release build are actually <quote>embarrassingly
        parallel</quote>.  Most of the tasks are very I/O intensive,
        so multiple high-speed disk drives is actually important that
        multiple processors in speeding up the <quote>make
        release</quote> process.  If multiple disks are used for
        different hierarchies in the <emphasis>chroot</emphasis>
        environment, then the CVS checkout of the ports and doc trees
        can be happening simultaneously to the <quote>make
        <literal>world</literal></quote> on another disk.  Using a
        <acronym>RAID</acronym> solution (hardware or software) can
        significantly decrease the overall build time.</para>
    </listitem>
      
    <listitem>
      <para><emphasis>Cross-building releases</emphasis> - Building
        IA-64 or Alpha release on x86 hardware? <quote>make
        TARGET=ia64 release</quote>.
      </para>
    </listitem>
    
    <listitem>
      <para><emphasis>Regression Testing</emphasis> - We need better
        automated correctness testing for FreeBSD.</para>
    </listitem>
      
    <listitem>
      <para><emphasis>Installation Tools</emphasis> - Our installation
        program has long since outlived its intended life span.
        Several projects are under development to provide a more
        advanced installation mechanism.  One of the most promising is
        the libh project[5] which aims to provided an intelligent new
        package framework and GUI installation program.</para>
    </listitem>
  </itemizedlist>
  
</sect1>

<!-- Acknowledgements -->
<sect1 id="ackno">
<title>Acknowledgements</title>

  <para>I would like to thank Jordan Hubbard for giving me the
    opportunity to take some on some of the release engineering
    responsibilities for FreeBSD 4.4 and also for all of his work
    throughout the years making FreeBSD what it is today.  Of course
    the release wouldn't have been possible without all of the
    release-related work done by &a.asami;, &a.steve;, &a.bmah;, &a.nik;,
    &a.obrien;, &a.kris;, &a.jhb; and the rest of the FreeBSD development
    community.  I would also like to thank &a.rgrimes;, &a.phk;, and others
    who worked on the release engineering tools in the very early days
    of FreeBSD.  This article was influenced by release engineering
    documents from the CSRG[14], the NetBSD Project[11], and John
    Baldwin's proposed release engineering process notes[12].</para>
</sect1>

<!-- Reference / Biblio Section -->
<sect1 id="biblio">
  <title>References</title>
  <para>[1] CVS - Concurrent Versions System 
  <ulink url="http://www.cvshome.org"></ulink></para>
  
  <para>[2] CVSup - The CVS-Optimized General Purpose Network File Distribution
  System <ulink url="http://www.polstra.com/projects/freeware/CVSup"></ulink>
  </para>
  
  <para>[3] <ulink url="http://bento.FreeBSD.org"></ulink></para>
  
  <para>[4] FreeBSD Ports Collection 
  <ulink url="http://www.FreeBSD.org/ports"></ulink></para>
  
  <para>[5] The libh Project 
  <ulink url="http://www.FreeBSD.org/projects/libh.html"></ulink></para>
  
  <para>[6] FreeBSD Committers <ulink 
  url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/contributors/staff-committers.html"></ulink>
  </para>
  
  <para>[7] FreeBSD Core-Team 
  <ulink url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/contributors/staff-core.html"></ulink></para>
  
  <para>[8] FreeBSD Handbook 
  <ulink url="http://www.FreeBSD.org/handbook"></ulink>
  </para>
  
  <para>[9] GNATS: The GNU Bug Tracking System
  <ulink url="http://www.gnu.org/software/gnats"></ulink>
  </para>
  
  <para>[10] FreeBSD PR Statistics
  <ulink url="http://www.FreeBSD.org/prstats/index.html"></ulink></para>
  
  <para>[11] NetBSD Developer Documentation: Release Engineering 
  <ulink url="http://www.NetBSD.org/developers/releng/index.html"></ulink>
  </para>
  
  <para>[12] John Baldwin's FreeBSD Release Engineering Proposal 
  <ulink url="http://people.FreeBSD.org/~jhb/docs/releng.txt"></ulink>
  </para>
  
  <para>[13] PXE Jumpstart Guide 
  <ulink 
  url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/pxe/index.html"></ulink>
  </para>
  
  <para>[14] Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic: 
  <ulink url="http://docs.freebsd.org/doc/papers/releng.html">
<emphasis>The Release Engineering of 4.3BSD</emphasis></ulink>
  </para>
</sect1>
</article>
