<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook XML V4.5-Based Extension//EN"
"../../../share/xml/freebsd45.dtd" [
<!ENTITY art.re.pkgs '<ulink url="&url.articles.releng-packages;/article.html">The Release Engineering of Third Party Packages</ulink>'>
]>
<article lang='en'>
<title>&os; Release Engineering</title>
<articleinfo>
<!-- This paper was presented at BSDCon Europe in Brighton, UK on
November 11, 2001. -->
<!-- The content in this paper was updated in March 2013 to
reflect the current FreeBSD Release process. -->
<confgroup>
<confdates>November 2001</confdates>
<conftitle>BSDCon Europe</conftitle>
</confgroup>
<authorgroup>
<author>
<firstname>Murray</firstname>
<surname>Stokely</surname>
<authorblurb>
<para>I've been involved in the development of &os; based products
since 1997 at Walnut Creek CDROM, BSDi, and now Wind River Systems.
&os; 4.4 was the first official release of &os; that I played
a significant part in.</para>
</authorblurb>
<affiliation>
<address><email>murray@FreeBSD.org</email>
<otheraddr><ulink url="http://www.FreeBSD.org/~murray/"></ulink></otheraddr>
</address>
</affiliation>
</author>
</authorgroup>
<legalnotice id="trademarks" role="trademarks">
&tm-attrib.freebsd;
&tm-attrib.cvsup;
&tm-attrib.intel;
&tm-attrib.general;
</legalnotice>
<pubdate>$FreeBSD$</pubdate>
<abstract>
<para>
<warning>
<para>2013/02/26: This document is partially outdated and does not
accurately describe the current release procedures of the
&os; Release Engineering team. The &os; Release
Engineering team is currently reviewing this document and
will publish updated content soon.
</para>
</warning>
</para>
<para>This paper describes the approach used by the &os;
release engineering team to make production quality releases
of the &os; Operating System. It details the methodology
used for the official &os; releases and describes the tools
available for those interested in producing customized &os;
releases for corporate rollouts or commercial
productization.</para>
</abstract>
</articleinfo>
<!-- Introduction -->
<sect1 id="introduction">
<title>Introduction</title>
<para>The development of &os; is a very open process. &os; is
comprised of contributions from thousands of people around the
world. The &os; Project provides
Subversion
<footnote>
<simpara>
Subversion, <ulink url="http://subversion.apache.org"></ulink>
</simpara>
</footnote>
access to the general public so that
others can have access to log messages, diffs (patches) between
development branches, and other productivity enhancements that
formal source code management provides. This has been a huge help
in attracting more talented developers to &os;. However, I
think everyone would agree that chaos would soon manifest if write
access to the main repository was opened up to everyone on the Internet.
Therefore only a <quote>select</quote> group of nearly 300 people are
given write access to the Subversion repository. These
<ulink url="&url.articles.contributors;/article.html#staff-committers">committers</ulink>
<footnote>
<simpara>
<ulink url="&url.articles.contributors;/article.html#staff-committers">FreeBSD committers</ulink>
</simpara>
</footnote>
are usually the people who do the bulk of &os; development. An elected
<ulink url="&url.base;/administration.html#t-core">Core Team</ulink>
<footnote>
<simpara>
<ulink url="&url.base;/administration.html#t-core">&os; Core Team</ulink>
</simpara>
</footnote>
of developers provide some level of direction over the project.</para>
<para>The rapid pace of <systemitem class="osname">&os;</systemitem>
development makes the main development branch unsuitable for the
everyday use by the general public. In particular, stabilizing
efforts are required for polishing the development system into a
production quality release. To solve this conflict, development
continues on several parallel tracks. The main development branch
is the <emphasis>HEAD</emphasis> or <emphasis>trunk</emphasis> of
our Subversion tree, known as <quote>&os;-CURRENT</quote> or
<quote>-CURRENT</quote> for short.</para>
<para>A set of more stable branches are maintained, known as
<quote>&os;-STABLE</quote> or <quote>-STABLE</quote> for short.
All branches live in a master Subversion repository maintained by the
&os; Project. &os;-CURRENT is the <quote>bleeding-edge</quote> of
&os; development where all new changes first enter the system.
&os;-STABLE is the development branch from which major releases
are made. Changes go into this branch at a different pace, and
with the general assumption that they have first gone into
&os;-CURRENT and have been thoroughly tested by our user
community.</para>
<para>The term <emphasis>stable</emphasis> in the name of the branch
refers to the presumed Application Binary Interface stability,
which is promised by the project. This means that a user
application compiled on an older version of the system from the
same branch works on a newer system from the same branch. The
ABI stability has improved greatly from the compared to previous
releases. In most cases, binaries from the older
<emphasis>STABLE</emphasis> systems run unmodified on newer systems,
including <emphasis>HEAD</emphasis>, assuming that the system
management interfaces are not used.</para>
<para>In the interim period between releases, monthly snapshots are
built automatically by the &os; Project build machines and made
available for download from <systemitem
class="resource">ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/</systemitem>.
The widespread availability of binary release snapshots, and the
tendency of our user community to keep up with -STABLE development
with Subversion and <quote><command>make</command>
<maketarget>buildworld</maketarget></quote>
<footnote>
<simpara>
<ulink url="&url.books.handbook;/makeworld.html">Rebuilding "world"</ulink>
</simpara>
</footnote>
helps to keep
&os;-STABLE in a very reliable condition even before the
quality assurance activities ramp up pending a major
release.</para>
<para>Bug reports and feature requests are continuously submitted by
users throughout the release cycle. Problems reports are entered into our
<application class="software">GNATS</application> database
<footnote>
<simpara>
GNATS: The GNU Bug Tracking System
<ulink url="http://www.gnu.org/software/gnats"></ulink>
</simpara>
</footnote>
through email, the &man.send-pr.1; application, or via the web
interface provided at <ulink
url="http://www.FreeBSD.org/send-pr.html"></ulink>.</para>
<para>To service our most conservative users, individual release
branches were introduced with &os; 4.3.
These release branches are created shortly before a final release
is made. After the release goes out, only the most critical
security fixes and additions are merged onto the release branch.
In addition to source updates via Subversion, binary patchkits are
available to keep systems on the
<emphasis>RELENG_<replaceable>X</replaceable>_<replaceable>Y</replaceable></emphasis>
branches updated.</para>
<sect2>
<title>What this article describes</title>
<para>The following sections of this article describe:</para>
<variablelist>
<varlistentry>
<term><xref linkend="release-proc"/></term>
<listitem>
<para>The different phases of the release engineering process
leading up to the actual system build.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="release-build"/></term>
<listitem>
<para>The actual build process.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="extensibility"/></term>
<listitem>
<para>How the base release may be extended by third parties.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="lessons-learned"/></term>
<listitem>
<para>Some of the lessons learned through the release of &os; 4.4.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="future"/></term>
<listitem>
<para>Future directions of development.</para>
</listitem>
</varlistentry>
</variablelist>
</sect2>
</sect1>
<!-- Release Process -->
<sect1 id="release-proc">
<title>Release Process</title>
<para>New releases of &os; are released from the -STABLE branch
at approximately four month intervals. The &os; release
process begins to ramp up 70-80 days before the anticipated release
date when the release engineer sends an email to the development
mailing lists to remind developers that they only have 15 days to
integrate new changes before the code freeze. During this time,
many developers perform what have become known as <quote>MFC
sweeps</quote>.</para>
<para><acronym>MFC</acronym> stands for <quote>Merge From
CURRENT</quote> and it describes the process of merging a tested
change from our -CURRENT development branch to our -STABLE branch.
Project policy requires any change to be first applied to
trunk, and merged to the -STABLE branches after sufficient
external testing was done by -CURRENT users (developers are
expected to extensively test the change before committing to
-CURRENT, but it is impossible for a person to exercise all usages
of the general-purpose operating system). Minimal MFC period is 3
days, which is typically used only for trivial or critical
bugfixes.</para>
<sect2>
<title>Code Review</title>
<para>Sixty days before the anticipated release, the source
repository enters a <quote>code freeze</quote>. During this
time, all commits to the -STABLE branch must be approved by the
&a.re;, the approval process is technically enforced by the
pre-commit hook. The kinds of changes that are allowed during
this period include:</para>
<itemizedlist>
<listitem>
<para>Bug fixes.</para>
</listitem>
<listitem>
<para>Documentation updates.</para>
</listitem>
<listitem>
<para>Security-related fixes of any kind.</para>
</listitem>
<listitem>
<para>Minor changes to device drivers, such as adding new Device
IDs.</para>
</listitem>
<listitem>
<para>Driver updates from the vendors.</para>
</listitem>
<listitem>
<para>Any additional change that the release engineering team feels
is justified, given the potential risk.</para>
</listitem>
</itemizedlist>
<para>Shortly after the code freeze is started, a
<emphasis>BETA1</emphasis> image is built and released for
widespread testing. During the code freeze, at least one beta
image or release candidate is released every two weeks until the
final release is ready. During the days preceeding the final
release, the release engineering team is in constant
communication with the security-officer team, the documentation
maintainers, and the port maintainers to ensure that all of the
different components required for a successful release are
available.</para>
<para>After the quality of the BETA images is satisfying enough,
and no large and potentially risky changes are planned, the
release branch is created and <emphasis>Release
Candidate</emphasis> (RC) images are built from the release
branch, instead of the BETA images from the STABLE branch.
Also, the freeze on the STABLE branch is lifted and release
branch enters a <quote>hard code freeze</quote> where it becomes
much harder to justify new changes to the system unless a
serious bug-fix or security issue is involved.</para>
</sect2>
<sect2>
<title>Final Release Checklist</title>
<para>When several BETA images have been made available for
widespread testing and all major issues have been resolved, the
final release <quote>polishing</quote> can begin.</para>
<sect3 id="rel-branch">
<title>Creating the Release Branch</title>
<note>
<para>In all examples below, <literal>$FSVN</literal>
refers to the location of the &os; Subversion repository,
<literal>svn+ssh://svn.freebsd.org/base/</literal>.</para>
</note>
<para>The layout of &os; branches in Subversion is
described in the <ulink url="&url.articles.committers-guide;/subversion-primer.html#subversion-primer-base-layout">Committer's Guide</ulink>.
The first step in creating a branch is to
identify the revision of the
<literal>stable/<replaceable>X</replaceable></literal> sources
that you want to branch <emphasis>from</emphasis>.</para>
<screen>&prompt.root; <userinput>svn log -v $FSVN/stable/9</userinput></screen>
<para>The next step is to create the <emphasis>release branch</emphasis>
</para>
<screen>&prompt.root; <userinput>svn cp $FSVN/stable/9@REVISION $FSVN/releng/9.2</userinput></screen>
<para>This branch can be checked out:</para>
<screen>&prompt.root; <userinput>svn co $FSVN/releng/9.2 src</userinput></screen>
<note>
<para>Creating the <literal>releng</literal> branch and
<literal>release</literal> tags is done by the <ulink
url="&url.base;/administration.html#t-re">Release
Engineering Team</ulink>.
</para>
</note>
<mediaobject>
<imageobject>
<imagedata fileref="branches-head" align="center"/>
</imageobject>
<textobject>
<phrase>&os; Development Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng3" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 3.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng4" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 4.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng5" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 5.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng6" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 6.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng7" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 7.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng8" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 8.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng9" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 9.x STABLE Branch</phrase>
</textobject>
</mediaobject>
</sect3>
<sect3 id="versionbump">
<title>Bumping up the Version Number</title>
<para>Before the final release can be tagged, built, and
released, the following files need to be modified to reflect
the correct version of &os;:</para>
<itemizedlist>
<listitem>
<para><filename>doc/en_US.ISO8859-1/books/handbook/mirrors/chapter.xml
</filename></para>
</listitem>
<listitem>
<para><filename>doc/en_US.ISO8859-1/books/porters-handbook/book.xml
</filename></para>
</listitem>
<listitem>
<para><filename>doc/share/xml/freebsd.ent</filename></para>
</listitem>
<listitem>
<para><filename>src/Makefile.inc1</filename></para>
</listitem>
<listitem>
<para><filename>src/UPDATING</filename></para>
</listitem>
<listitem>
<para><filename>src/gnu/usr.bin/groff/tmac/mdoc.local</filename></para>
</listitem>
<listitem>
<para><filename>src/release/Makefile</filename></para>
</listitem>
<listitem>
<para><filename>src/release/doc/en_US.ISO8859-1/share/xml/release.dsl</filename></para>
</listitem>
<listitem>
<para><filename>src/release/doc/share/examples/Makefile.relnotesng</filename></para>
</listitem>
<listitem>
<para><filename>src/release/doc/share/xml/release.ent</filename></para>
</listitem>
<listitem>
<para><filename>src/share/examples/cvsup/standard-supfile</filename></para>
</listitem>
<listitem>
<para><filename>src/sys/conf/newvers.sh</filename></para>
</listitem>
<listitem>
<para><filename>src/sys/sys/param.h</filename></para>
</listitem>
<listitem>
<para><filename>src/usr.sbin/pkg_install/add/main.c</filename></para>
</listitem>
<listitem>
<para><filename>www/en/docs/man.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/en/cgi/ports.cgi</filename></para>
</listitem>
<listitem>
<para><filename>ports/Tools/scripts/release/config</filename></para>
</listitem>
</itemizedlist>
<para>The release notes and errata files also need to be adjusted for the
new release (on the release branch) and truncated appropriately
(on the stable/current branch):</para>
<itemizedlist>
<listitem>
<para><filename>src/release/doc/en_US.ISO8859-1/relnotes/common/new.xml
</filename></para>
</listitem>
<listitem>
<para><filename>src/release/doc/en_US.ISO8859-1/errata/article.xml
</filename></para>
</listitem>
</itemizedlist>
<para><application>Sysinstall</application> should be updated to note
the number of available ports and the amount of disk space required
for the Ports Collection.
<footnote>
<simpara>
&os; Ports Collection
<ulink url="http://www.FreeBSD.org/ports"></ulink>
</simpara>
</footnote>
This information is currently kept in
<filename>src/usr.sbin/sysinstall/dist.c</filename>.</para>
<para>After the release has been built, a number of file should
be updated to announce the release to the world.</para>
<itemizedlist>
<listitem>
<para><filename>doc/share/images/articles/releng/branches-releng<replaceable>X</replaceable>.pic</filename></para>
</listitem>
<listitem>
<para><filename>www/share/xml/advisories.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/share/xml/includes.release.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/share/xml/includes.release.xsl</filename></para>
</listitem>
<listitem>
<para><filename>www/en/releases/*</filename></para>
</listitem>
<listitem>
<para><filename>www/en/releng/index.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/en/news/news.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/en/search/web.atoz</filename></para>
</listitem>
<listitem>
<para><filename>src/share/misc/bsd-family-tree</filename></para>
</listitem>
</itemizedlist>
</sect3>
<sect3>
<title>Creating the Release Tag</title>
<para>When the final release is ready, the following command
will create the <literal>release/9.2.0</literal>
tag.</para>
<screen>&prompt.root; <userinput>svn cp $FSVN/releng/9.2 $FSVN/release/9.2.0</userinput></screen>
<para>The Documentation and Ports managers are responsible for
tagging their respective trees with the <literal>tags/RELEASE_9_2_0</literal>
tag.</para>
<sidebar>
<para>When the Subversion <command>svn cp</command> command
is used to create a <emphasis>release tag</emphasis>,
this identifies the source at a specific point in time.
By creating tags, we ensure that future release builders
will always be able to use the exact same source we used to create the
official &os; Project releases.</para>
</sidebar>
</sect3>
</sect2>
</sect1>
<!-- Release Building -->
<sect1 id="release-build">
<title>Release Building</title>
<para>&os; <quote>releases</quote> can be built by anyone with a
fast machine and access to a source repository. (That should be
everyone, since we offer Subversion access !
See the
<ulink url="&url.books.handbook;/svn.html">Subversion section
in the Handbook</ulink> for
details.) The <emphasis>only</emphasis> special requirement is
that the &man.md.4; device must be available. If the
device is not loaded into your kernel, then the kernel module
should be automatically loaded when &man.mdconfig.8; is executed
during the boot media creation phase. All of the tools necessary
to build a release are available from the Subversion repository in
<filename>src/release</filename>. These tools aim to provide a
consistent way to build &os; releases. A complete release can
actually be built with only a single command, including the
creation of <acronym>ISO</acronym> images suitable for burning to
CDROM or DVD, and an FTP install directory. &man.release.7; fully
documents the <command>src/release/generate-release.sh</command>
script which is used to build a release. <command>generate-release.sh</command>
is a wrapper around the Makefile target: <command>make release</command>.</para>
<sect2>
<title>Building a Release</title>
<para>&man.release.7; documents the exact commands required to
build a &os; release. The following sequences of commands can build
an 9.2.0 release:</para>
<screen>&prompt.root; <userinput>cd /usr/src/release</userinput></screen>
<screen>&prompt.root; <userinput>sh generate-release.sh release/9.2.0 /local3/release</userinput></screen>
<para>After running these commands, all prepared release
files are available in <filename>/local3/release/R</filename>
directory.</para>
<para>The release <filename>Makefile</filename> can be broken down into several distinct
steps.</para>
<itemizedlist>
<listitem>
<para>Creation of a sanitized system environment in a separate
directory hierarchy with <quote><command>make
<literal>installworld</literal></command></quote>.
</para>
</listitem>
<listitem>
<para>Checkout from Subversion of a clean version of the system source,
documentation, and ports into the release build hierarchy.</para>
</listitem>
<listitem>
<para>Population of <filename>/etc</filename> and
<filename>/dev</filename> in the chrooted
environment.</para>
</listitem>
<listitem>
<para>chroot into the release build hierarchy, to make it harder for
the outside environment to taint this build.</para>
</listitem>
<listitem>
<para><command>make world</command>
in the chrooted environment.</para>
</listitem>
<listitem>
<para>Build of Kerberos-related binaries.</para>
</listitem>
<listitem>
<para>Build <filename>GENERIC</filename> kernel.</para>
</listitem>
<listitem>
<para>Creation of a staging directory tree where the binary
distributions will be built and packaged.</para>
</listitem>
<listitem>
<para>Build and installation of the documentation toolchain needed to
convert the documentation source (SGML) into HTML and text documents
that will accompany the release.</para>
</listitem>
<listitem>
<para>Build and installation of the actual documentation
(user manuals, tutorials, release notes, hardware compatibility lists,
and so on.)</para>
</listitem>
<listitem>
<para>Package up distribution tarballs of the binaries and sources.
</para>
</listitem>
<listitem>
<para>Create FTP installation hierarchy.</para>
</listitem>
<listitem>
<para><emphasis>(optionally)</emphasis> Create ISO images for
CDROM/DVD media.</para>
</listitem>
</itemizedlist>
<para>For more information about the release build infrastructure,
please see &man.release.7;.</para>
<note><para>It is important to remove any site-specific settings
from <filename>/etc/make.conf</filename>. For example, it would
be unwise to distribute binaries that were built on a system
with <varname>CPUTYPE</varname> set to a specific
processor.</para></note>
</sect2>
<sect2>
<title>Contributed Software (<quote>ports</quote>)</title>
<para>The <ulink url="http://www.FreeBSD.org/ports">&os; Ports
collection</ulink> is a collection of over &os.numports;
third-party software packages available for &os;. The &a.portmgr;
is responsible for maintaining a consistent ports tree that can be used
to create the binary packages that accompany official &os;
releases.</para>
<para>The release engineering activities for our collection of
third-party packages is beyond the scope of this document. A
separate article, &art.re.pkgs;, covers this topic
in depth.</para>
</sect2>
<sect2>
<title>Release ISOs</title>
<para>Starting with &os; 4.4, the &os; Project decided to
release all four ISO images that were previously sold on the
<emphasis>BSDi/Wind River Systems/FreeBSD Mall</emphasis>
<quote>official</quote> CDROM distributions. Each of the four
discs must contain a <filename>README.TXT</filename> file that
explains the contents of the disc, a
<filename>CDROM.INF</filename> file that provides meta-data for
the disc so that &man.sysinstall.8; can validate and use the
contents, and a <filename>filename.txt</filename> file that
provides a manifest for the disc. This
<emphasis>manifest</emphasis> can be created with a simple
command:</para>
<screen>/stage/cdrom&prompt.root; <userinput>find . -type f | sed -e 's/^\.\///' | sort > filename.txt</userinput></screen>
<para>The specific requirements of each CD are outlined below.</para>
<sect3>
<title>Disc 1</title>
<para>The first disc is almost completely created by
<command>make
release</command>. The only changes
that should be made to the <filename>disc1</filename> directory are the addition of
a <filename>tools</filename> directory, and as many popular
third party software packages as will fit on the disc. The
<filename>tools</filename> directory contains software that allow users to create
installation floppies from other operating systems. This disc
should be made bootable so that users of modern PCs do not
need to create installation floppy disks.</para>
<para>If a custom kernel of &os; is to be included, then
&man.sysinstall.8; and &man.release.7; must be updated to
include installation instructions. The relevant code is contained
in <filename>src/release</filename> and <filename>src/usr.sbin/sysinstall</filename>.
Specifically, the file <filename>src/release/Makefile</filename>, and
<filename>dist.c</filename>, <filename>dist.h</filename>,
<filename>menus.c</filename>, <filename>install.c</filename>, and
<filename>Makefile</filename> will need to be updated under
<filename>src/usr.sbin/sysinstall</filename>. Optionally, you may choose
to update <filename>sysinstall.8</filename>.</para>
</sect3>
<sect3>
<title>Disc 2</title>
<para>The second disc is also largely created by <command>make
release</command>. This disc contains a <quote>live
filesystem</quote> that can be used from &man.sysinstall.8; to
troubleshoot a &os; installation. This disc should be
bootable and should also contain a compressed copy of the CVS
repository in the <filename>CVSROOT</filename> directory and
commercial software demos in the <filename>commerce</filename>
directory.</para>
</sect3>
<sect3>
<title>Discs 3 and 4</title>
<para>The remaining two discs contain additional software
packages for &os;. The packages should be clustered so that
a package and all of its <emphasis>dependencies</emphasis> are
included on the same disc. More information about the
creation of these discs is provided in the &art.re.pkgs;
article.</para>
</sect3>
<sect3>
<title>Multi-volume support</title>
<para><application>Sysinstall</application> supports multiple
volume package installations. This requires that each disc
have an <filename>INDEX</filename> file containing all of the
packages on all volumes of a set, along with an extra field
that indicates which volume that particular package is on.
Each volume in the set must also have the
<literal>CD_VOLUME</literal> variable set in the
<filename>cdrom.inf</filename> file so that sysinstall can
tell which volume is which. When a user attempts to install a
package that is not on the current disc, sysinstall will
prompt the user to insert the appropriate one.</para>
</sect3>
</sect2>
</sect1>
<!-- Distribution -->
<sect1 id="distribution">
<title>Distribution</title>
<sect2 id="dist-ftp">
<title>FTP Sites</title>
<para>When the release has been thoroughly tested and packaged for
distribution, the master FTP site must be updated. The official
&os; public FTP sites are all mirrors of a master server that
is open only to other FTP sites. This site is known as
<hostid>ftp-master</hostid>. When the release is ready, the
following files must be modified on <hostid>ftp-master</hostid>:</para>
<variablelist>
<varlistentry>
<term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/</filename></term>
<listitem>
<para>The installable FTP directory as output from <command>make
release</command>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><filename>/pub/FreeBSD/ports/<replaceable>arch</replaceable>/packages-<replaceable>X.Y</replaceable>-release/</filename></term>
<listitem><para>The complete package build for this
release.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/tools</filename></term>
<listitem><para>A symlink to
<filename>../../../tools</filename>.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/packages</filename></term>
<listitem><para>A symlink to
<filename>../../../ports/<replaceable>arch</replaceable>/packages-<replaceable>X.Y</replaceable>-release</filename>.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/ISO-IMAGES/<replaceable>X.Y</replaceable>/<replaceable>X.Y</replaceable>-RELEASE-<replaceable>arch</replaceable>-*.iso</filename></term>
<listitem><para>The ISO images. The <quote>*</quote> is
<filename>disc1</filename>, <filename>disc2</filename>, etc.
Only if there is a <filename>disc1</filename> and there is an
alternative first installation CD (for example a
stripped-down install with no windowing system) there may
be a <filename>mini</filename> as well.</para>
</listitem>
</varlistentry>
</variablelist>
<para>For more information about the distribution mirror
architecture of the &os; FTP sites, please see the <ulink
url="&url.articles.hubs;/">Mirroring &os;</ulink> article.</para>
<para>It may take many hours to two days after updating
<hostid>ftp-master</hostid> before a majority of the Tier-1 FTP
sites have the new software depending on whether or not a package
set got loaded at the same time. It is imperative that the release
engineers coordinate with the &a.mirror-announce; before announcing the general
availability of new software on the FTP sites. Ideally
the release package set should be loaded at least four
days prior to release day. The release bits should be
loaded between 24 and 48 hours before the planned release
time with <quote>other</quote> file permissions turned off.
This will allow the mirror sites to download it but the
general public will not be able to download it from the mirror
sites. Mail should be sent to &a.mirror-announce; at the time
the release bits get posted saying the release has been staged
and giving the time that the mirror sites should begin allowing
access. Be sure to include a time zone with the
time, for example make it relative to GMT.</para>
</sect2>
<sect2 id="dist-cdrom">
<title>CD-ROM Replication</title>
<para>Coming soon: Tips for sending &os; ISOs to a replicator
and quality assurance measures to be taken.</para>
</sect2>
</sect1>
<!-- Extensibility -->
<sect1 id="extensibility">
<title>Extensibility</title>
<para>Although &os; forms a complete operating system, there is
nothing that forces you to use the system exactly as we have
packaged it up for distribution. We have tried to design the
system to be as extensible as possible so that it can serve as a
platform that other commercial products can be built on top
of. The only <quote>rule</quote> we have about this is that if you
are going to distribute &os; with non-trivial changes, we
encourage you to document your enhancements! The &os; community
can only help support users of the software we provide. We
certainly encourage innovation in the form of advanced
installation and administration tools, for example, but we cannot
be expected to answer questions about it.</para>
<sect2>
<title>Scripting <command>sysinstall</command></title>
<para>The &os; system installation and configuration tool,
&man.sysinstall.8;, can be scripted to provide automated installs
for large sites. This functionality can be used in conjunction
with &intel; PXE
<footnote>
<simpara>
<ulink url="&url.books.handbook;/network-pxe-nfs.html"></ulink>
</simpara>
</footnote>
to bootstrap systems from the network.
</para>
</sect2>
</sect1>
<!-- Lessons Learned -->
<sect1 id="lessons-learned">
<title>Lessons Learned from &os; 4.4</title>
<para>The release engineering process for 4.4 formally began on
August 1st, 2001. After that date all commits to the
<literal>RELENG_4</literal> branch of &os; had to be explicitly
approved by the &a.re;. The first
release candidate for the x86 architecture was released on August
16, followed by 4 more release candidates leading up to the final
release on September 18th. The security officer was very involved
in the last week of the process as several security issues were
found in the earlier release candidates. A total of over
<emphasis>500</emphasis> emails were sent to the &a.re; in
little over a month.</para>
<para>Our user community has made it very clear that the security
and stability of a &os; release should not be sacrificed for
any self-imposed deadlines or target release dates. The &os;
Project has grown tremendously over its lifetime and the need for
standardized release engineering procedures has never been more
apparent. This will become even more important as &os; is
ported to new platforms.</para>
</sect1>
<!-- Future Directions -->
<sect1 id="future">
<title>Future Directions</title>
<para>It is imperative for our release engineering activities to
scale with our growing userbase. Along these lines we are working
very hard to document the procedures involved in producing &os;
releases.</para>
<itemizedlist>
<listitem>
<para><emphasis>Parallelism</emphasis> - Certain portions of the
release build are actually <quote>embarrassingly
parallel</quote>. Most of the tasks are very I/O intensive,
so having multiple high-speed disk drives is actually more important than
using multiple processors in speeding up the <command>make
release</command> process. If multiple disks are used for
different hierarchies in the &man.chroot.2;
environment, then the CVS checkout of the <filename>ports</filename> and <filename>doc</filename> trees
can be happening simultaneously as the <command>make
world</command> on another disk. Using a
<acronym>RAID</acronym> solution (hardware or software) can
significantly decrease the overall build time.</para>
</listitem>
<listitem>
<para><emphasis>Cross-building releases</emphasis> - Building
IA-64 or Alpha release on x86 hardware? <command>make
TARGET=ia64 release</command>.
</para>
</listitem>
<listitem>
<para><emphasis>Regression Testing</emphasis> - We need better
automated correctness testing for &os;.</para>
</listitem>
<listitem>
<para><emphasis>Installation Tools</emphasis> - Our installation
program has long since outlived its intended life span.
Several projects are under development to provide a more
advanced installation mechanism. The libh project was one
such project that aimed to provide an intelligent new package
framework and GUI installation program.</para>
</listitem>
</itemizedlist>
</sect1>
<!-- Acknowledgements -->
<sect1 id="ackno">
<title>Acknowledgements</title>
<para>I would like to thank Jordan Hubbard for giving me the
opportunity to take on some of the release engineering
responsibilities for &os; 4.4 and also for all of his work
throughout the years making &os; what it is today. Of course
the release would not have been possible without all of the
release-related work done by &a.asami;, &a.steve;, &a.bmah;, &a.nik;,
&a.obrien;, &a.kris;, &a.jhb; and the rest of the &os; development
community. I would also like to thank &a.rgrimes;, &a.phk;, and others
who worked on the release engineering tools in the very early days
of &os;. This article was influenced by release engineering
documents from the CSRG
<footnote>
<simpara>
Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic:
<ulink url="http://docs.FreeBSD.org/44doc/papers/releng.html">
<emphasis>The Release Engineering of 4.3BSD</emphasis></ulink>
</simpara>
</footnote>
,
the NetBSD Project ,
<footnote>
<simpara>
NetBSD Developer Documentation: Release Engineering
<ulink url="http://www.NetBSD.org/developers/releng/index.html"></ulink>
</simpara>
</footnote>
, and John
Baldwin's proposed release engineering process notes.
<footnote>
<simpara>
John Baldwin's &os; Release Engineering Proposal
<ulink url="http://people.FreeBSD.org/~jhb/docs/releng.txt"></ulink>
</simpara>
</footnote>
</para>
</sect1>
</article>