os/doc
3
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Add an article on the Release Engineering / QA activities for the

Browse source 
third party software packages that accompany an official FreeBSD
release.

This document currently describes the "package split" procedure that
Steve Price uses to create clusters of like packages with similar
dependencies that are grouped onto specific discs.

In the future, this article should document the scripts that generate
clean packages on the bento cluster, the steps necessary to setup
another group of machines as a "Ports Cluster", and the scripts that
are used to verify the integrity of a package split.

Submitted by:	   steve
Sponsored by:	   FreeBSD Mall, Inc.
This commit is contained in:
Murray Stokely 2002-02-25 14:33:46 +00:00
parent bf6eeb13b4
commit ef2065d02e
Notes: svn2git 2020-12-08 03:00:23 +00:00 
svn path=/head/; revision=12282
3 changed files with 404 additions and 0 deletions
en_US.ISO8859-1/articles/releng-packages
Makefilearticle.sgmlextra.css

20
en_US.ISO8859-1/articles/releng-packages/Makefile Normal file
View file

@ -0,0 +1,20 @@
#
# $FreeBSD$
#
# Article: FreeBSD Release Engineering of Third Party Software Packages
DOC?= article
FORMATS?= html
INSTALL_COMPRESSED?= gz
INSTALL_ONLY_COMPRESSED?=
# SGML content
SRCS= article.sgml
CSS_SHEET_ADDITIONS= extra.css
DOC_PREFIX?= ${.CURDIR}/../../..
.include "${DOC_PREFIX}/share/mk/doc.project.mk"

368
en_US.ISO8859-1/articles/releng-packages/article.sgml Normal file
View file

@ -0,0 +1,368 @@
<!--
The FreeBSD Documentation Project
-->
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook V4.1-Based Extension//EN" [
<!ENTITY % man PUBLIC "-//FreeBSD//ENTITIES DocBook Manual Page Entities//EN">
%man;
<!ENTITY % freebsd PUBLIC "-//FreeBSD//ENTITIES DocBook Miscellaneous FreeBSD Entities//EN">
%freebsd;
<!ENTITY % authors PUBLIC "-//FreeBSD//ENTITIES DocBook Author Entities//EN">
%authors;
]>
<article>
<title>FreeBSD Release Engineering for Third Party Software
Packages</title>
<articleinfo>
<authorgroup>
<author>
<firstname>Steve</firstname>
<surname>Price</surname>
<affiliation>
<address><email>steve@FreeBSD.org</email></address>
</affiliation>
</author>
</authorgroup>
<pubdate>$FreeBSD$</pubdate>
<abstract>
<para>This paper describes the approach used by the FreeBSD
release engineering team to produce a high quality package set
suitable for official FreeBSD release media. This document is
a work in progress, but eventually it will cover the process
used to build a clean package set on the FreeBSD.org "Ports
Cluster", how to configure any other set of machines as a
ports cluster, how to split up the packages for the release
media, and how to verify that a package set is
consistent.</para>
</abstract>
</articleinfo>
<!-- Introduction
<sect1 id="introduction">
<title>Introduction</title>
<para><emphasis>Coming Soon</emphasis></para>
</sect1>
-->
<sect1 id="portbuild">
<title>Building packages from the Ports Collection</title>
<para>The <ulink url="http://www.FreeBSD.org/ports">FreeBSD Ports
collection</ulink> is a collection of over &os.numports;
third-party software packages available for FreeBSD. The ports
team (portmgr@FreeBSD.org) is responsible for maintaining a
consistent ports tree that can be used to create the binary
packages that accompany a given FreeBSD release.</para>
<sect2>
<title>The Ports Cluster</title>
<para>In order to provide a consistent set of third-party
packages for FreeBSD releases, every port is built in a
separate chroot environment, starting with an empty
<filename>/usr/local</filename> and
<filename>/usr/X11R6</filename>. The requisite dependencies
are installed as packages before the build proceeds. This
enforces <emphasis>consistency</emphasis> in the package build
process. By starting the package build in a pristine
environment, we can assure that the package metadata (such as
required dependencies) is accurate, and so we will never
generate packages that might work on some systems and not on
others depending on what software was previously
installed.</para>
<para>The <quote>Ports Cluster</quote> for the x86 architecture
currently consists of a master node (Dual Pentium III 733Mhz)
and 8 slave nodes (Pentium III 800Mhz) to do the actual
package builds. With this configuration, a complete package
build takes over 24 hours. These machines are co-located with
the other FreeBSD Project equipment at Yahoo's corner of
Exodus in Santa Clara, CA.</para>
<para>The <quote>Ports Cluster</quote> for the Alpha
architecture consists of 7 PWS 500A machines donated by Compaq
and also co-located with Yahoo's facilities.</para>
</sect2>
</sect1>
<sect1>
<title>The Package Split</title>
<para>For FreeBSD 4.4 over 4.1 gigabytes of packages were created.
This causes a problem for CDROM distributions because we would
like to ship as many packages as possible without making the
user insert another disc to satisfy dependencies. The solution
is to create <quote>clusters</quote> of like packages with
similar dependencies and group these onto specific discs. This
section describes the software and methodology used to create
those package sets for the official FreeBSD release
discs.</para>
<para>First off you'll need to get a copy of the tarball from the
following URL :</para>
<para><ulink
url="http://people.freebsd.org/~steve/release-scripts.tgz"></ulink></para>
<para>Copy thisc archive to a machine that has enough free HD
space to hold 2 to 3 times the size of the package set that you
wish to split. The tarball will extract into the current
working directory so make sure you've created one suitably named
directory for the release you are working on.</para>
<para>After you've extracted the files, you will notice the
following files :</para>
<variablelist>
<varlistentry>
<term><filename>config</filename></term>
<listitem><para>This file contains the free space on each disc
and whether packages, distfiles, or both are allowed on any
given disc. The first column is the disc name. It must be
of the form disc[0-9a-z]. Currently it is setup to allow
for 10 discs (4 for the release set and 6 for the toolkit).
There's an implied extra disc called scratch where all of
the remaining distfiles/packages land if they don't fit
elsewhere. The second column can be either a 1 or 0 where 1
says that it is okay to place packages on this disc. The
third column works the same way except that it controls
whether distfiles are placed on this disc. The last column
denotes the number of bytes of free space on a
disc.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>doit.sh</filename></term>
<listitem><para>This is the workhorse. Once you have all the
files in place and things properly configured this script
directs the process of splitting packages. Beware it is
interactive so you need to keep an eye on it as it runs.
More details on what happens in this script will
follow.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>scripts/checkdeps.pl</filename></term>
<listitem><para>Makes sure all packages dependencies are
satisfied given an INDEX file and a directory of
packages.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>scripts/oneshot.pl</filename></term>
<listitem><para>This is where all the magic (and I use that
term loosely as it is mostly just a brute force approach)
happens. Given a list of required packages for each disc
and a set of packages/distfiles this is the script that
places a package or distfile on a disc along with all of its
dependencies.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>scripts/print-cdrom-packages.sh</filename></term>
<listitem><para>This file is a copy of
<filename>src/release/scripts/print-cdrom-packages.sh</filename>
from the release you are working on.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>scripts/scrubindex.pl</filename></term>
<listitem><para>This script removes lines from an INDEX file
for packages that aren't present. It also removes the
XFree86 dependencies. NOTE: you'll need to tweak the value
of the xdep variable to make sure the version number is
correct.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>scripts/setup.sh</filename></term>
<listitem><para>This is a helper script that I use on the
bento cluster to grab a copy of the ports tree and the
matching set of the packages/distfiles.</para></listitem>
</varlistentry>
</variablelist>
<para>Here's a checklist of things you'll need to check or
configure before going any further.</para>
<orderedlist>
<listitem><para>Edit <filename>config</filename> to denote the
number of discs you have, their sizes, and whether you want
them want to contain packages, distfiles, both, or
neither.</para></listitem>
<listitem><para>Make sure you remove the gen directory if
there's an old one laying around. This directory contains
working files that will only be valid for the current
split.</para></listitem>
<listitem><para>On your first pass through a split it is best to
fake the copying of packages and distfiles. This will save
both time and diskspace while you do a couple of trial runs to
make sure things fit, etc. In the
<filename>scripts/oneshot.pl</filename> set the fake variable
to 1 and instead of actually copying the files it will
&man.touch.1; them. Be sure you turn this off or set fake to
0 before you give the resultant discs to the person that will
be mastering the discs otherwise they'll get a directory full
of zero-sized files.</para></listitem>
<listitem><para>Make sure you have a recent copy of the
<filename>print-cdrom-packages.sh</filename> and that it is
from the correct release.</para></listitem>
<listitem><para>Check to make sure the XFree86 dependency in
<filename>scripts/scrubindex.pl</filename> has the correct
version number. You'll also need to make sure this value is
correct in <filename>doit.sh</filename> as
well.</para></listitem>
</orderedlist>
<para>Next you'll need to get a copy of the ports tree, packages,
and distfiles from a recent build on the package cluster. See
the <filename>scripts/setup.sh</filename> for a working example
but essentially here's what needs to be done.</para>
<orderedlist>
<listitem><para>Grab a copy of <filename>ports.tar.gz</filename>
and extract it into the ports directory alongside
<filename>doit.sh</filename> and the
<Filename>scripts</filename> directory.</para></listitem>
<listitem><para>Remove the packages/distfiles directories or
symlinks. Bento has these as symlinks and you'll have mixed
results if you don't get rid of them before
proceeding.</para></listitem>
<listitem><para>Create a new ports/packages directory and copy
the package set from the package building
cluster.</para></listitem>
<listitem><para>Create a new ports/distfiles directory and copy
the distfiles from the package building cluster. NOTE: if you
don't want any distfiles simply create the directory and leave
it empty. This directory must be present even if it doesn't
contain anything.</para></listitem>
</orderedlist>
<para>Now we're finally ready for the fun task of actually
splitting the packages. You start the processing by running
<command>./doit.sh</command>. The first time you run it here's
what it does.</para>
<orderedlist>
<listitem><para>Create a list of the restricted (can't be on the
master FTP site) ports.</para></listitem>
<listitem><para>Asks you if you'd like to remove the restricted
ports. Most of the time you'll want to answer (y)es
here.</para></listitem>
<listitem><para>Create a list of the packages/distfiles that
can't be put on the discs.</para></listitem>
<listitem><para>Asks you if you'd like to remove the
non-cdromable packages/distfiles. Most of the time you'll
want to answer (y)es here.</para></listitem>
<listitem><para>Copies the INDEX from the ports directory to the
gen directory. In doing so it removes the lines for ports
where the packages don't exist. It also checks to make sure
that all of the required dependency packages are
present.</para></listitem>
<listitem><para>Create a list of packages that are required on
each disc.</para></listitem>
<listitem><para>Asks you if you'd like to populate the discs.
After populating each disc it will check for missing
dependencies, scrub the INDEX file, and create the
CHECKSUM.MD5 file.</para></listitem>
<listitem><para>Check to make sure the required packages made it
on each disc and gives you a summary of the sizes of each
disc.</para></listitem>
</orderedlist>
<para>After going through this the first time if you are lucky
enough that all of the required packages built and fit on each
disc. All you need to do is set fake to 0 in
<Filename>scripts/oneshot.pl</filename> and re-run
<command>./doit.sh</command>. The second and subsequent times
around it will skip steps 1-5 above. If you want to re-run any
of those steps refer to doit.sh for which files need to be
removed to not short-circuit those steps. If you want to repeat
all of these steps then the easiest way is to <command>rm -rf
gen</command>.</para>
<para>Upon successful completion the packages/distfiles will be in
the <filename>disc*</filename> directories and the leftover will
be in the scratch directory.</para>
<para>What to do if things go wrong? Here's some common gotchas
and workarounds.</para>
<variablelist>
<varlistentry>
<term>Missing required packages</term>
<listitem><para>This is a pretty common occurrence. You'll
either need to wait for a new set of packages where the
missing packages were built or get someone to re-start the
package build for you. DO NOT attempt to build the missing
packages on your own machine and add them into the fray.
While you might be able to get away with this if you are
extremely careful the vast majority of the time you'll miss
some little detail and the simple process of adding a
package could make hundreds of others come up mysteriously
broken.</para></listitem>
</varlistentry>
<varlistentry>
<term>Required packages won't fit</term>
<listitem><para>This happens on occasion too and is relatively
easy to fix. Simply edit print-cdrom-packages.sh to move
packages around until they fit. Yes this is an iterative
process and one of the reasons why you should enable fake in
<filename>scripts/oneshot.pl</filename> until you've gotten
things the way you want them. Re-run
<command>./doit.sh</command> after you made your
adjustments.</para></listitem>
</varlistentry>
<varlistentry>
<term>Required packages not on the right (or any) disc</term>
<listitem><para>This usually means you didn't add them to
<filename>print-cdrom-packages.sh</filename> or you put them
on the wrong disc. This script is the gospel by which this
whole process determines where a package must be. If you
want to force a package to land on a particular disc this is
the only way to ensure that it will
happen.</para></listitem>
</varlistentry>
</variablelist>
<para>If you get completely stuck and can't figure out why things
are borked or how to fix them then email &a.steve; for
assistance.</para>
</sect1>
</article>

16
en_US.ISO8859-1/articles/releng-packages/extra.css Normal file
View file

@ -0,0 +1,16 @@
/*
* Netscape 4 does not recognice the @import directive of CSS, so we
* can't add an additional stylesheet layer on top of the default one.
* Instead, we use this hack to copy this file to the end of
* docbook.css.
*
* $FreeBSD$
*/
/* @import "docbook.css"; */
/* Customization that looks good for this particular article. */
DIV.TITLEPAGE {
text-align: center;
}