diff --git a/en_US.ISO8859-1/articles/releng-packages/Makefile b/en_US.ISO8859-1/articles/releng-packages/Makefile new file mode 100644 index 0000000000..00cedcd8e7 --- /dev/null +++ b/en_US.ISO8859-1/articles/releng-packages/Makefile @@ -0,0 +1,20 @@ +# +# $FreeBSD$ +# +# Article: FreeBSD Release Engineering of Third Party Software Packages + +DOC?= article + +FORMATS?= html + +INSTALL_COMPRESSED?= gz +INSTALL_ONLY_COMPRESSED?= + +# SGML content +SRCS= article.sgml + +CSS_SHEET_ADDITIONS= extra.css + +DOC_PREFIX?= ${.CURDIR}/../../.. + +.include "${DOC_PREFIX}/share/mk/doc.project.mk" diff --git a/en_US.ISO8859-1/articles/releng-packages/article.sgml b/en_US.ISO8859-1/articles/releng-packages/article.sgml new file mode 100644 index 0000000000..ef4636c263 --- /dev/null +++ b/en_US.ISO8859-1/articles/releng-packages/article.sgml @@ -0,0 +1,368 @@ +<!-- + The FreeBSD Documentation Project +--> + +<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook V4.1-Based Extension//EN" [ +<!ENTITY % man PUBLIC "-//FreeBSD//ENTITIES DocBook Manual Page Entities//EN"> +%man; +<!ENTITY % freebsd PUBLIC "-//FreeBSD//ENTITIES DocBook Miscellaneous FreeBSD Entities//EN"> +%freebsd; +<!ENTITY % authors PUBLIC "-//FreeBSD//ENTITIES DocBook Author Entities//EN"> +%authors; +]> + +<article> + <title>FreeBSD Release Engineering for Third Party Software + Packages</title> + <articleinfo> + <authorgroup> + <author> + <firstname>Steve</firstname> + <surname>Price</surname> + <affiliation> + <address><email>steve@FreeBSD.org</email></address> + </affiliation> + </author> + </authorgroup> + + <pubdate>$FreeBSD$</pubdate> + + <abstract> + <para>This paper describes the approach used by the FreeBSD + release engineering team to produce a high quality package set + suitable for official FreeBSD release media. This document is + a work in progress, but eventually it will cover the process + used to build a clean package set on the FreeBSD.org "Ports + Cluster", how to configure any other set of machines as a + ports cluster, how to split up the packages for the release + media, and how to verify that a package set is + consistent.</para> + </abstract> + + </articleinfo> + + <!-- Introduction + + <sect1 id="introduction"> + <title>Introduction</title> + + <para><emphasis>Coming Soon</emphasis></para> + + </sect1> + +--> + <sect1 id="portbuild"> + <title>Building packages from the Ports Collection</title> + + <para>The <ulink url="http://www.FreeBSD.org/ports">FreeBSD Ports + collection</ulink> is a collection of over &os.numports; + third-party software packages available for FreeBSD. The ports + team (portmgr@FreeBSD.org) is responsible for maintaining a + consistent ports tree that can be used to create the binary + packages that accompany a given FreeBSD release.</para> + + <sect2> + <title>The Ports Cluster</title> + + <para>In order to provide a consistent set of third-party + packages for FreeBSD releases, every port is built in a + separate chroot environment, starting with an empty + <filename>/usr/local</filename> and + <filename>/usr/X11R6</filename>. The requisite dependencies + are installed as packages before the build proceeds. This + enforces <emphasis>consistency</emphasis> in the package build + process. By starting the package build in a pristine + environment, we can assure that the package metadata (such as + required dependencies) is accurate, and so we will never + generate packages that might work on some systems and not on + others depending on what software was previously + installed.</para> + + <para>The <quote>Ports Cluster</quote> for the x86 architecture + currently consists of a master node (Dual Pentium III 733Mhz) + and 8 slave nodes (Pentium III 800Mhz) to do the actual + package builds. With this configuration, a complete package + build takes over 24 hours. These machines are co-located with + the other FreeBSD Project equipment at Yahoo's corner of + Exodus in Santa Clara, CA.</para> + + <para>The <quote>Ports Cluster</quote> for the Alpha + architecture consists of 7 PWS 500A machines donated by Compaq + and also co-located with Yahoo's facilities.</para> + </sect2> + </sect1> + + <sect1> + <title>The Package Split</title> + + <para>For FreeBSD 4.4 over 4.1 gigabytes of packages were created. + This causes a problem for CDROM distributions because we would + like to ship as many packages as possible without making the + user insert another disc to satisfy dependencies. The solution + is to create <quote>clusters</quote> of like packages with + similar dependencies and group these onto specific discs. This + section describes the software and methodology used to create + those package sets for the official FreeBSD release + discs.</para> + + <para>First off you'll need to get a copy of the tarball from the + following URL :</para> + + <para><ulink + url="http://people.freebsd.org/~steve/release-scripts.tgz"></ulink></para> + + <para>Copy thisc archive to a machine that has enough free HD + space to hold 2 to 3 times the size of the package set that you + wish to split. The tarball will extract into the current + working directory so make sure you've created one suitably named + directory for the release you are working on.</para> + + <para>After you've extracted the files, you will notice the + following files :</para> + + <variablelist> + <varlistentry> + <term><filename>config</filename></term> + + <listitem><para>This file contains the free space on each disc + and whether packages, distfiles, or both are allowed on any + given disc. The first column is the disc name. It must be + of the form disc[0-9a-z]. Currently it is setup to allow + for 10 discs (4 for the release set and 6 for the toolkit). + There's an implied extra disc called scratch where all of + the remaining distfiles/packages land if they don't fit + elsewhere. The second column can be either a 1 or 0 where 1 + says that it is okay to place packages on this disc. The + third column works the same way except that it controls + whether distfiles are placed on this disc. The last column + denotes the number of bytes of free space on a + disc.</para></listitem> + </varlistentry> + + <varlistentry> + <term><filename>doit.sh</filename></term> + + <listitem><para>This is the workhorse. Once you have all the + files in place and things properly configured this script + directs the process of splitting packages. Beware it is + interactive so you need to keep an eye on it as it runs. + More details on what happens in this script will + follow.</para></listitem> + </varlistentry> + + <varlistentry> + <term><filename>scripts/checkdeps.pl</filename></term> + + <listitem><para>Makes sure all packages dependencies are + satisfied given an INDEX file and a directory of + packages.</para></listitem> + </varlistentry> + + <varlistentry> + <term><filename>scripts/oneshot.pl</filename></term> + + <listitem><para>This is where all the magic (and I use that + term loosely as it is mostly just a brute force approach) + happens. Given a list of required packages for each disc + and a set of packages/distfiles this is the script that + places a package or distfile on a disc along with all of its + dependencies.</para></listitem> + </varlistentry> + + <varlistentry> + <term><filename>scripts/print-cdrom-packages.sh</filename></term> + + <listitem><para>This file is a copy of + <filename>src/release/scripts/print-cdrom-packages.sh</filename> + from the release you are working on.</para></listitem> + </varlistentry> + + <varlistentry> + <term><filename>scripts/scrubindex.pl</filename></term> + + <listitem><para>This script removes lines from an INDEX file + for packages that aren't present. It also removes the + XFree86 dependencies. NOTE: you'll need to tweak the value + of the xdep variable to make sure the version number is + correct.</para></listitem> + </varlistentry> + + <varlistentry> + <term><filename>scripts/setup.sh</filename></term> + + <listitem><para>This is a helper script that I use on the + bento cluster to grab a copy of the ports tree and the + matching set of the packages/distfiles.</para></listitem> + </varlistentry> + </variablelist> + + <para>Here's a checklist of things you'll need to check or + configure before going any further.</para> + + <orderedlist> + <listitem><para>Edit <filename>config</filename> to denote the + number of discs you have, their sizes, and whether you want + them want to contain packages, distfiles, both, or + neither.</para></listitem> + + <listitem><para>Make sure you remove the gen directory if + there's an old one laying around. This directory contains + working files that will only be valid for the current + split.</para></listitem> + + <listitem><para>On your first pass through a split it is best to + fake the copying of packages and distfiles. This will save + both time and diskspace while you do a couple of trial runs to + make sure things fit, etc. In the + <filename>scripts/oneshot.pl</filename> set the fake variable + to 1 and instead of actually copying the files it will + &man.touch.1; them. Be sure you turn this off or set fake to + 0 before you give the resultant discs to the person that will + be mastering the discs otherwise they'll get a directory full + of zero-sized files.</para></listitem> + + <listitem><para>Make sure you have a recent copy of the + <filename>print-cdrom-packages.sh</filename> and that it is + from the correct release.</para></listitem> + + <listitem><para>Check to make sure the XFree86 dependency in + <filename>scripts/scrubindex.pl</filename> has the correct + version number. You'll also need to make sure this value is + correct in <filename>doit.sh</filename> as + well.</para></listitem> + </orderedlist> + + <para>Next you'll need to get a copy of the ports tree, packages, + and distfiles from a recent build on the package cluster. See + the <filename>scripts/setup.sh</filename> for a working example + but essentially here's what needs to be done.</para> + + <orderedlist> + + <listitem><para>Grab a copy of <filename>ports.tar.gz</filename> + and extract it into the ports directory alongside + <filename>doit.sh</filename> and the + <Filename>scripts</filename> directory.</para></listitem> + + <listitem><para>Remove the packages/distfiles directories or + symlinks. Bento has these as symlinks and you'll have mixed + results if you don't get rid of them before + proceeding.</para></listitem> + + <listitem><para>Create a new ports/packages directory and copy + the package set from the package building + cluster.</para></listitem> + + <listitem><para>Create a new ports/distfiles directory and copy + the distfiles from the package building cluster. NOTE: if you + don't want any distfiles simply create the directory and leave + it empty. This directory must be present even if it doesn't + contain anything.</para></listitem> + </orderedlist> + + <para>Now we're finally ready for the fun task of actually + splitting the packages. You start the processing by running + <command>./doit.sh</command>. The first time you run it here's + what it does.</para> + + <orderedlist> + <listitem><para>Create a list of the restricted (can't be on the + master FTP site) ports.</para></listitem> + + <listitem><para>Asks you if you'd like to remove the restricted + ports. Most of the time you'll want to answer (y)es + here.</para></listitem> + + <listitem><para>Create a list of the packages/distfiles that + can't be put on the discs.</para></listitem> + + <listitem><para>Asks you if you'd like to remove the + non-cdromable packages/distfiles. Most of the time you'll + want to answer (y)es here.</para></listitem> + + <listitem><para>Copies the INDEX from the ports directory to the + gen directory. In doing so it removes the lines for ports + where the packages don't exist. It also checks to make sure + that all of the required dependency packages are + present.</para></listitem> + + <listitem><para>Create a list of packages that are required on + each disc.</para></listitem> + + <listitem><para>Asks you if you'd like to populate the discs. + After populating each disc it will check for missing + dependencies, scrub the INDEX file, and create the + CHECKSUM.MD5 file.</para></listitem> + + <listitem><para>Check to make sure the required packages made it + on each disc and gives you a summary of the sizes of each + disc.</para></listitem> + </orderedlist> + + <para>After going through this the first time if you are lucky + enough that all of the required packages built and fit on each + disc. All you need to do is set fake to 0 in + <Filename>scripts/oneshot.pl</filename> and re-run + <command>./doit.sh</command>. The second and subsequent times + around it will skip steps 1-5 above. If you want to re-run any + of those steps refer to doit.sh for which files need to be + removed to not short-circuit those steps. If you want to repeat + all of these steps then the easiest way is to <command>rm -rf + gen</command>.</para> + + <para>Upon successful completion the packages/distfiles will be in + the <filename>disc*</filename> directories and the leftover will + be in the scratch directory.</para> + + <para>What to do if things go wrong? Here's some common gotchas + and workarounds.</para> + + <variablelist> + <varlistentry> + <term>Missing required packages</term> + + <listitem><para>This is a pretty common occurrence. You'll + either need to wait for a new set of packages where the + missing packages were built or get someone to re-start the + package build for you. DO NOT attempt to build the missing + packages on your own machine and add them into the fray. + While you might be able to get away with this if you are + extremely careful the vast majority of the time you'll miss + some little detail and the simple process of adding a + package could make hundreds of others come up mysteriously + broken.</para></listitem> + </varlistentry> + + <varlistentry> + <term>Required packages won't fit</term> + + <listitem><para>This happens on occasion too and is relatively + easy to fix. Simply edit print-cdrom-packages.sh to move + packages around until they fit. Yes this is an iterative + process and one of the reasons why you should enable fake in + <filename>scripts/oneshot.pl</filename> until you've gotten + things the way you want them. Re-run + <command>./doit.sh</command> after you made your + adjustments.</para></listitem> + </varlistentry> + + <varlistentry> + <term>Required packages not on the right (or any) disc</term> + + <listitem><para>This usually means you didn't add them to + <filename>print-cdrom-packages.sh</filename> or you put them + on the wrong disc. This script is the gospel by which this + whole process determines where a package must be. If you + want to force a package to land on a particular disc this is + the only way to ensure that it will + happen.</para></listitem> + </varlistentry> + </variablelist> + + <para>If you get completely stuck and can't figure out why things + are borked or how to fix them then email &a.steve; for + assistance.</para> + + </sect1> + +</article> diff --git a/en_US.ISO8859-1/articles/releng-packages/extra.css b/en_US.ISO8859-1/articles/releng-packages/extra.css new file mode 100644 index 0000000000..7cf2be1d7d --- /dev/null +++ b/en_US.ISO8859-1/articles/releng-packages/extra.css @@ -0,0 +1,16 @@ +/* + * Netscape 4 does not recognice the @import directive of CSS, so we + * can't add an additional stylesheet layer on top of the default one. + * Instead, we use this hack to copy this file to the end of + * docbook.css. + * + * $FreeBSD$ + */ + +/* @import "docbook.css"; */ + +/* Customization that looks good for this particular article. */ + +DIV.TITLEPAGE { + text-align: center; +}