359 lines
15 KiB
XML
359 lines
15 KiB
XML
<?xml version="1.0" encoding="iso-8859-1"?>
|
|
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook XML V5.0-Based Extension//EN"
|
|
"http://www.FreeBSD.org/XML/share/xml/freebsd50.dtd">
|
|
<article xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en">
|
|
|
|
<info><title>FreeBSD Release Engineering for Third Party Software
|
|
Packages</title>
|
|
<authorgroup>
|
|
<author><personname><firstname>Steve</firstname><surname>Price</surname></personname><affiliation>
|
|
<address><email>steve@FreeBSD.org</email></address>
|
|
</affiliation></author>
|
|
</authorgroup>
|
|
|
|
<legalnotice xml:id="trademarks" role="trademarks">
|
|
&tm-attrib.freebsd;
|
|
&tm-attrib.intel;
|
|
&tm-attrib.xfree86;
|
|
&tm-attrib.general;
|
|
</legalnotice>
|
|
|
|
<pubdate>$FreeBSD$</pubdate>
|
|
|
|
<releaseinfo>$FreeBSD$</releaseinfo>
|
|
|
|
<abstract>
|
|
<para>This paper describes the approach used by the FreeBSD
|
|
ports management team to produce a high quality package set
|
|
suitable for official FreeBSD release media. This document is
|
|
a work in progress, but eventually it will cover the process
|
|
used to build a clean package set on the FreeBSD.org <quote>Ports
|
|
Cluster</quote>, how to configure any other set of machines as a
|
|
ports cluster, how to split up the packages for the release
|
|
media, and how to verify that a package set is
|
|
consistent.</para>
|
|
</abstract>
|
|
|
|
</info>
|
|
|
|
<!-- Introduction
|
|
|
|
<sect1 id="introduction">
|
|
<title>Introduction</title>
|
|
|
|
<para><emphasis>Coming Soon</emphasis></para>
|
|
|
|
</sect1>
|
|
|
|
-->
|
|
<sect1 xml:id="portbuild">
|
|
<title>Building packages from the Ports Collection</title>
|
|
|
|
<para>The <link xlink:href="http://www.FreeBSD.org/ports">FreeBSD Ports
|
|
collection</link> is a collection of over &os.numports;
|
|
third-party software packages available for FreeBSD. The &a.portmgr;
|
|
is responsible for maintaining a consistent ports tree that can be used
|
|
to create the binary packages that accompany a given FreeBSD
|
|
release.</para>
|
|
|
|
<sect2>
|
|
<title>The Ports Cluster</title>
|
|
|
|
<para>In order to provide a consistent set of third-party
|
|
packages for FreeBSD releases, every port is built in a
|
|
separate chroot environment, starting with an empty
|
|
<filename>/usr/local</filename> and
|
|
<filename>/usr/X11R6</filename>. The requisite dependencies
|
|
are installed as packages before the build proceeds. This
|
|
enforces <emphasis>consistency</emphasis> in the package build
|
|
process. By starting the package build in a pristine
|
|
environment, we can assure that the package metadata (such as
|
|
required dependencies) is accurate. This way, we will never
|
|
generate packages that might work on some systems and not on
|
|
others depending on what software was previously
|
|
installed.</para>
|
|
|
|
<para>The <quote>Ports Cluster</quote> for the x86 architecture
|
|
currently consists of a master node (Dual &pentium; III 733MHz)
|
|
and 8 slave nodes (&pentium; III 800MHz) to do the actual
|
|
package builds. With this configuration, a complete package
|
|
build takes over 24 hours. These machines are co-located with
|
|
the other FreeBSD Project equipment at Yahoo's corner of
|
|
Exodus in Santa Clara, CA.</para>
|
|
|
|
<para>The <quote>Ports Cluster</quote> for the Alpha
|
|
architecture consists of 7 PWS 500A machines donated by Compaq
|
|
and also co-located with Yahoo's facilities.</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>The Package Split</title>
|
|
|
|
<para>For FreeBSD 4.4 over 4.1 gigabytes of packages were created.
|
|
This causes a problem for CDROM distributions because we would
|
|
like to ship as many packages as possible without making the
|
|
user insert another disc to satisfy dependencies. The solution
|
|
is to create <quote>clusters</quote> of like packages with
|
|
similar dependencies and group these onto specific discs. This
|
|
section describes the software and methodology used to create
|
|
those package sets for the official FreeBSD release
|
|
discs.</para>
|
|
|
|
<para>The scripts and other files needed to produce a package
|
|
split can be found in the CVS tree in
|
|
<filename>ports/Tools/scripts/release</filename>.
|
|
Copy this directory to a machine that has enough free disk
|
|
space to hold 2 to 3 times the size of the package set that you
|
|
wish to split.</para>
|
|
|
|
<para>The following scripts are present in this directory:</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term><filename>config</filename></term>
|
|
|
|
<listitem><para>This file contains the free space on each disc
|
|
and whether packages, distfiles, or both are allowed on any
|
|
given disc. The first column is the disc name. It must be
|
|
of the form <literal>disc[0-9a-z]</literal>. Currently it is set up
|
|
to allow for 10 discs (4 for the release set and 6 for the toolkit).
|
|
There is an implied extra disc called <quote>scratch</quote> where
|
|
all of the remaining distfiles/packages land if they do not fit
|
|
elsewhere. The second column can be either a 1 or 0, where 1
|
|
says that it is okay to place packages on this disc. The
|
|
third column works the same way, but it controls
|
|
whether distfiles are placed on this disc. The last column
|
|
denotes the number of bytes of free space on a
|
|
disc.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>doit.sh</filename></term>
|
|
|
|
<listitem><para>This is the workhorse. Once you have all the
|
|
files in place and things properly configured this script
|
|
directs the process of splitting packages. Beware it is
|
|
interactive so you need to keep an eye on it as it runs.
|
|
More details on what happens in this script will
|
|
follow.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>checkdeps.pl</filename></term>
|
|
|
|
<listitem><para>Makes sure all packages dependencies are
|
|
satisfied given an <filename>INDEX</filename> file and a directory
|
|
of packages.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>oneshot.pl</filename></term>
|
|
|
|
<listitem><para>This is where all the magic (and I use that
|
|
term loosely as it is mostly just a brute force approach)
|
|
happens. Given a list of required packages for each disc
|
|
and a set of packages/distfiles this is the script that
|
|
places a package or distfile on a disc along with all of its
|
|
dependencies.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>print-cdrom-packages.sh</filename></term>
|
|
|
|
<listitem><para>This file is a copy of
|
|
<filename>src/release/scripts/print-cdrom-packages.sh</filename>
|
|
from the release you are working on.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>scrubindex.pl</filename></term>
|
|
|
|
<listitem><para>This script removes lines from an
|
|
<filename>INDEX</filename> file for packages that are not present.
|
|
It also removes the &xfree86; dependencies. NOTE: you will need to
|
|
tweak the value of the <varname>xdep</varname> variable to make sure
|
|
the version number is correct.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><filename>setup.sh</filename></term>
|
|
|
|
<listitem><para>This is a helper script that I use on the
|
|
ports building cluster to grab a copy of the ports tree and the
|
|
matching set of the packages/distfiles.</para></listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<para>Here is a checklist of things you will need to check or
|
|
configure before going any further.</para>
|
|
|
|
<orderedlist>
|
|
<listitem><para>Edit <filename>config</filename> to denote the
|
|
number of discs you have, their sizes, and whether you want
|
|
them want to contain packages, distfiles, both, or
|
|
neither.</para></listitem>
|
|
|
|
<listitem><para>Make sure you remove the <varname>gen</varname>
|
|
directory if there is an old one laying around. This directory
|
|
contains working files that will only be valid for the current
|
|
split.</para></listitem>
|
|
|
|
<listitem><para>On your first pass through a split it is best to
|
|
fake the copying of packages and distfiles. This will save
|
|
both time and disk space while you do a couple of trial runs to
|
|
make sure things fit, etc. In the
|
|
<filename>oneshot.pl</filename> set the <varname>fake</varname>
|
|
variable to 1 and instead of actually copying the files it will
|
|
&man.touch.1; them. Be sure you turn this off or set
|
|
<varname>fake</varname> to 0 before you give the resultant discs to
|
|
the person that will be mastering the discs otherwise they will get a
|
|
directory full of zero-sized files.</para></listitem>
|
|
|
|
<listitem><para>Make sure you have a recent copy of the
|
|
<filename>print-cdrom-packages.sh</filename> and that it is
|
|
from the correct release.</para></listitem>
|
|
|
|
<listitem><para>Check to make sure the &xfree86; dependency in
|
|
<filename>scrubindex.pl</filename> has the correct
|
|
version number. You will also need to make sure this value is
|
|
correct in <filename>doit.sh</filename> as
|
|
well.</para></listitem>
|
|
</orderedlist>
|
|
|
|
<para>Next you will need to get a copy of the ports tree, packages,
|
|
and distfiles from a recent build on the package cluster. See
|
|
the <filename>setup.sh</filename> for a working example
|
|
but essentially here is what needs to be done.</para>
|
|
|
|
<orderedlist>
|
|
|
|
<listitem><para>Grab a copy of <filename>ports.tar.gz</filename>
|
|
and extract it into the <filename>ports</filename> directory alongside
|
|
<filename>doit.sh</filename> and the
|
|
<filename>scripts</filename> directory.</para></listitem>
|
|
|
|
<listitem><para>Remove the packages/distfiles directories or
|
|
symlinks. Bento has these as symlinks and you will have mixed
|
|
results if you do not get rid of them before
|
|
proceeding.</para></listitem>
|
|
|
|
<listitem><para>Create a new ports/packages directory and copy
|
|
the package set from the package building
|
|
cluster.</para></listitem>
|
|
|
|
<listitem><para>Create a new ports/distfiles directory and copy
|
|
the distfiles from the package building cluster. NOTE: if you
|
|
do not want any distfiles simply create the directory and leave
|
|
it empty. This directory must be present even if it does not
|
|
contain anything.</para></listitem>
|
|
</orderedlist>
|
|
|
|
<para>Now we are finally ready for the fun task of actually
|
|
splitting the packages. You start the processing by running
|
|
<command>./doit.sh</command>. Here is what it does the first
|
|
time you run it.</para>
|
|
|
|
<orderedlist>
|
|
<listitem><para>Create a list of the restricted (can not be on the
|
|
master FTP site) ports.</para></listitem>
|
|
|
|
<listitem><para>Asks you if you would like to remove the restricted
|
|
ports. Most of the time you will want to answer (y)es
|
|
here.</para></listitem>
|
|
|
|
<listitem><para>Create a list of the packages/distfiles that
|
|
can not be put on the discs.</para></listitem>
|
|
|
|
<listitem><para>Asks you if you would like to remove the
|
|
non-cdromable packages/distfiles. Most of the time you will
|
|
want to answer (y)es here.</para></listitem>
|
|
|
|
<listitem><para>Copies the <filename>INDEX</filename> from the
|
|
<filename>ports</filename> directory to the <filename>gen</filename>
|
|
directory. In doing so it removes the lines for ports
|
|
where the packages do not exist. It also checks to make sure
|
|
that all of the required dependency packages are
|
|
present.</para></listitem>
|
|
|
|
<listitem><para>Create a list of packages that are required on
|
|
each disc.</para></listitem>
|
|
|
|
<listitem><para>Asks you if you would like to populate the discs.
|
|
After populating each disc it will check for missing
|
|
dependencies, scrub the <filename>INDEX</filename> file, and create the
|
|
<filename>CHECKSUM.MD5</filename> file.</para></listitem>
|
|
|
|
<listitem><para>Check to make sure the required packages made it
|
|
on each disc and gives you a summary of the sizes of each
|
|
disc.</para></listitem>
|
|
</orderedlist>
|
|
|
|
<para>After going through this the first time if you are lucky
|
|
enough that all of the required packages built and fit on each
|
|
disc. All you need to do is set <varname>fake</varname> to 0 in
|
|
<filename>oneshot.pl</filename> and re-run
|
|
<command>./doit.sh</command>. The second and subsequent times
|
|
around it will skip steps 1-5 above. If you want to re-run any
|
|
of those steps refer to <filename>doit.sh</filename> for which files
|
|
need to be removed to not short-circuit those steps. If you want to
|
|
repeat all of these steps then the easiest way is to <command>rm -rf
|
|
gen</command>.</para>
|
|
|
|
<para>Upon successful completion the packages/distfiles will be in
|
|
the <filename>disc*</filename> directories and the leftover will
|
|
be in the <filename>scratch</filename> directory.</para>
|
|
|
|
<para>What to do if things go wrong? Here is some common gotchas
|
|
and workarounds.</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>Missing required packages</term>
|
|
|
|
<listitem><para>This is a pretty common occurrence. You will
|
|
either need to wait for a new set of packages where the
|
|
missing packages were built or get someone to re-start the
|
|
package build for you. <emphasis>Do not</emphasis> attempt to build
|
|
the missing packages on your own machine and add them into the fray.
|
|
While you might be able to get away with this if you are
|
|
extremely careful the vast majority of the time you will miss
|
|
some little detail and the simple process of adding a
|
|
package could make hundreds of others come up mysteriously
|
|
broken.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Required packages will not fit</term>
|
|
|
|
<listitem><para>This happens on occasion too and is relatively
|
|
easy to fix. Simply edit
|
|
<filename>print-cdrom-packages.sh</filename> to move
|
|
packages around until they fit. Yes this is an iterative
|
|
process and one of the reasons why you should enable
|
|
<varname>fake</varname> in <filename>oneshot.pl</filename> until you
|
|
have gotten things the way you want them. Re-run
|
|
<command>./doit.sh</command> after you made your
|
|
adjustments.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Required packages not on the right (or any) disc</term>
|
|
|
|
<listitem><para>This usually means you did not add them to
|
|
<filename>print-cdrom-packages.sh</filename> or you put them
|
|
on the wrong disc. This script is the gospel by which this
|
|
whole process determines where a package must be. If you
|
|
want to force a package to land on a particular disc this is
|
|
the only way to ensure that it will
|
|
happen.</para></listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<para>If you get completely stuck and can not figure out why things
|
|
are borked or how to fix them then email &a.steve.email; for
|
|
assistance.</para>
|
|
|
|
</sect1>
|
|
|
|
</article>
|