# Package Building Procedures

The &os; Ports Management Team
$FreeBSD$
2003
2004
The &os; Ports Management Team
&tm-attrib.freebsd; &tm-attrib.intel; &tm-attrib.sparc; &tm-attrib.general;

## Introduction and Conventions

In order to provide pre-compiled binaries of third-party applications for &os;, the ports collection is regularly built on one of the Package Building Clusters. Currently, there are two such clusters: pointyhat.FreeBSD.org and dosirak.kr.FreeBSD.org.

Most of the package building magic occurs under the `/var/portbuild` directory. Unless otherwise specified, all paths will be relative to this location. ${arch} will be used to specify one of the package architectures (&i386;, alpha, &sparc64;, ia64, and amd64), and ${branch} will be used to specify the build branch (4, 5, 4-exp).

## Build Client Management

The &i386;, alpha, amd64, and two &sparc64; clients currently netboot from pointyhat; the other sparc64 client and ia64 clients are self-hosted. In all cases they set themselves up at boot-time to prepare to build packages.

In the latest round of portbuild updates, disconnected cluster node support has been added. A disconnected node is one that does not mount the cluster master via NFS. It could be a remote node, for example. The cluster master rsync's the interesting data (ports, src, and doc trees, bindist tarballs, scripts, etc.) to disconnected nodes during the node-setup phase. Then, the disconnected portbuild directory is nullfs-mounted for chroot builds.

The ports-${arch} user can &man.ssh.1; as root onto each of the ${arch} nodes.

The `scripts/allgohans` script can be used to run a command on all of the ${arch} clients.

The `scripts/checkmachines` script is used to monitor the load on all the nodes of the build cluster, and schedule which nodes build which ports. This script is not very robust, and has a tendency to die. It is best to start up this script on the build master (either pointyhat or dosirak) after boot time using a &man.while.1; loop.

## Chroot Build Environment Setup

Package builds are performed in a `chroot` populated by the `portbuild` script using the `${arch}/${branch}/tarballs/bindist.tar` file. This tarball is created by the `mkbindist` script which reads the `${arch}/${branch}/mkbindist.conf` file to decide how to create the tarball.

The script should be run as root with the following command:

    /var/portbuild&prompt.root; scripts/mkbindist ${arch} ${branch}

If `ftp=1` in `mkbindist.conf` then a pre-built release will be downloaded via FTP from the location specified by ftp://${ftpserver}/${ftpurl}/${rel}. If `ftp=0` and `buildworld=1` then `mkbindist` will call `makeworld` to build a new world [`XXX` This is currently broken].

If both `ftp=0` and `buildworld=0` then `mkbindist` will use the pre-existing contents of ${worlddir} to create `bindist.tar`. In practice this means that you must have already installed a world in ${worlddir}, which is typically installed with the `makeworld` script:

    /var/portbuild&prompt.root; scripts/makeworld ${arch} ${branch} [-nocvs]

This command builds a world from the `${arch}/${branch}/src` tree and installs it into ${worlddir}. The tree will be updated first unless `-nocvs` is specified.

The `bindist.tar` file is extracted onto each client at client boot time, and at the start of each pass of the `dopackages` script.

## Starting the Build

The `scripts/dopackages*` scripts are used to perform the builds. Most useful are:

- `dopackages.5` - Perform a 5.X build

- `dopackages.4` - Perform a 4.X build

- `dopackages.4-exp` - Perform a 4.X build with experimental patches (4-exp branch)

These are wrappers around `dopackages`, and are all symlinked to `dopackages.wrapper`. New branch wrapper scripts can be created by symlinking `dopackages.${branch}` to `dopackages.wrapper`. These scripts take a number of arguments. For example:

    dopackages.5 ${arch} [-options]

`[-options]` may be zero or more of the following:

- `-nofinish` - Do not perform post-processing once the build is complete. Useful if you expect that the build will need to be restarted once it finishes. This option should always be used for normal build operations.

- `-finish` - Perform post-processing only.

- `-restart` - Restart an interrupted (or non-`finish`ed) build from the beginning. Ports that failed on the previous build will be rebuilt.

- `-continue` - Restart an interrupted (or non-`finish`ed) build. Will not rebuild ports that failed on the previous build.

- `-incremental` - Compare the interesting fields of the new `INDEX` with the previous one, remove packages and log files for the old ports that have changed, and rebuild the rest. This cuts down on build times substantially since unchanged ports do not get rebuilt everytime.

- `-cdrom` - This package build is intended to end up on a CD-ROM, so `NO_CDROM` packages and distfiles should be deleted in post-processing.

- `-nobuild` - Perform all the preprocessing steps, but do not actually do the package build.

- `-noindex` - Do not rebuild `INDEX` during preprocessing.

- `-noduds` - Do not rebuild the `duds` file (ports that are never built, e.g. those marked `IGNORE`, `NO_PACKAGE`, etc.) during preprocessing.

- `-trybroken` - Try to build `BROKEN` ports (off by default because the &i386; cluster is fast enough now that when doing incremental builds, more time was spent rebuilding things that were going to fail anyway. Conversely, the other clusters are slow enough that it would be a waste of time to try and build `BROKEN` ports.

- `-nocvs` - Do not `cvs update` the `src` tree during preprocessing.

- `-noportscvs` - Do not `cvs update` the `ports` tree during preprocessing.

- `-nodoccvs` - Do not `cvs update` the `doc` tree during preprocessing.

- `-norestr` - Do not attempt to build `RESTRICTED` ports.

- `-plistcheck` - Make it fatal for ports to leave behind files after deinstallation.

- `-distfiles` - Collect distfiles that pass `make checksum` for later uploading to ftp-master. Use this sparingly because it takes up a lot of disk space. You should remove the distfiles once they have been transfered to ftp-master.

- `-fetch-original` - Fetch the distfile from the original `MASTER_SITES` rather than ftp-master.

Make sure the ${arch} build is run as the ports-${arch} user or it will complain loudly.

The actual package build itself occurs in two identical phases. The reason for this is that sometimes transient problems (e.g. NFS failures, FTP sites being unreachable, etc.) may halt the build. Doing things in two phases is a workaround for these types of problems.

Be careful that `ports/Makefile` does not specify any empty subdirectories. This is especially important if you are doing a 4-exp build. If the build process encounters an empty subdirectory, both package build phases will stop short, and an error similar to the following will be written to `${arch}/${branch}/make.[0|1]`:

    don't know how to make dns-all(continuing)

To correct this problem, simply comment out or remove the `SUBDIR` entries that point to empty subdirectories. After doing this, you can restart the build by running the proper `dopackages` command with the `-restart` option.

## Anatomy of a Build

A full build without any `-no` options performs the following operations in the specified order:

1. A CVS checkout of the current `ports` tree [*]

2. A CVS checkout of the running branch's `doc` tree [*]

3. A CVS checkout of the running branch's `src` tree [*]

4. Checks which ports do not have a `SUBDIR` entry in their respective category's `Makefile` [*]

5. Creates the `duds` file, which is a list of ports not to build [*] [+]

6. Generates a fresh `INDEX` file [*] [+]

7. Sets up the nodes that will be used in the build [*] [+]

8. Builds a list of restricted ports [*] [+]

9. Builds packages (phase 1) [++]

10. Performs another node setup [+]

11. Builds packages (phase 2) [++]

[*] Status of these steps can be found in `${arch}/${branch}/build.log` as well as on stderr of the tty running the `dopackages` command.

[+] If any of these steps fail, the build will stop cold in its tracks.

[++] Status of these steps can be found in `${arch}/${branch}/make.[0|1]`, where `make.0` is the log file used by phase 1 of the package build and `make.1` is the log file used by phase 2. Individual ports will write their build logs to `${arch}/${branch}/logs` and their error logs to `${arch}/${branch}/errors`.

## Interrupting a Build

Sending a `HUP` signal to the `dopackages*` shell processes or to any `make` process invoked by those scripts is usually sufficient to interrupt the build. The package builds dispatched by `make` to the client machines will clean themselves up after a few minutes (check with `ps x` until they all go away). The following command usually does the trick:

    &prompt.user; killall -HUP sh ssh make

Remove the `${arch}/lock` file before trying to restart the build.

## Monitoring the Build

The `scripts/stats ${branch}` command counts the number of packages currently built.

Running `cat /var/portbuild/*/loads/*` shows the client loads and number of concurrent builds in progress.

Running `tail -f ${arch}/${branch}/build.log` shows the overall build progress.

If a build is failing, and it is not immediately obvious from the port build log as to why, you can preserve the `WRKDIR` for further analysis. To do this, touch a file called `.keep` in the port's directory. The next time the cluster tries to build this port, it will tar, compress, and copy the `WRKDIR` to `${arch}/${branch}/wrkdirs`.

Keep an eye on &man.df.1; output. If the `/var/portbuild` file system becomes full then Bad Things happen.

## Release Builds

When building packages for a release, it may be necessary to manually update the `ports` and `src` trees to the release tag and use `-nocvs` and `-noportscvs`.

To build package sets intended for use on a CD-ROM, use the `-cdrom` option to `dopackages`.

Assuming disk space is available on the cluster, use `-distfiles` to collect distfiles.

You must run the initial build with `-distfiles` to collect all the fetched distfiles.

After the initial build completes, restart the build with `-restart -distfiles -fetch-original` to collect updated distfiles as well. Then, once the build is post-processed, take an inventory of the list of files fetched:

    &prompt.user; cd ${arch}/${branch}
    &prompt.user; find distfiles > distfiles-${release}

This inventory file typically lives in `i386/${branch}` on the cluster master.

This is useful to aid in periodically cleaning out the distfiles from ftp-master. When space gets tight, distfiles from recent releases can be kept while others can be thrown away.

Once the distfiles have been uploaded (see below), the final release package set must be created. Just to be on the safe side, run the `${arch}/${branch}/cdrom.sh` script by hand to make sure all the CD-ROM restricted packages and distfiles have been pruned. Then, copy the `${arch}/${branch}/packages` directory to `${arch}/${branch}/packages-${release}`. Once the packages are safely moved off, contact the &a.re; and inform them of the release package location.

Remember to coordinate with the &a.re; about the timing and status of the release builds.

## Uploading Packages

Once a build has completed, packages and/or distfiles can be transferred to ftp-master for propagation to the FTP mirror network. If the build was run with `-nofinish`, then make sure to follow up with `dopackages -finish` to post-process the packages (removes `RESTRICTED` and `NO_CDROM` packages where appropriate, prunes packages not listed in `INDEX`, removes from `INDEX` references to packages not built, and generates a `CHECKSUM.MD5` summary); and distfiles (moves them from the temporary `distfiles/.pbtmp` directory into `distfiles/` and removes `RESTRICTED` and `NO_CDROM` distfiles).

It is usually a good idea to run the `restricted.sh` and/or `cdrom.sh` scripts by hand after `dopackages` finishes just to be safe. Run the `restricted.sh` script before uploading to ftp-master, then run `cdrom.sh` before preparing the final package set for a release.

Packages can be copied to the staging area on ftp-master with something like the following:

    &prompt.root; cd /var/portbuild/${arch}/${branch}
    &prompt.root; tar cfv - packages/ | ssh portmgr@ftp-master tar xfC - w/ports/${arch}/tmp/${branch}

Then log into ftp-master, verify that the package set was transferred successfully, remove the package set that the new package set is to replace (in `~/w/ports/${arch}`), and move the new set into place.

Some of the directories on ftp-master are, in fact, symlinks. Be sure you move the new packages directory over the real destination directory, and not one of the symlinks that points to it.

Distfiles can be transferred via `rsync`:

    &prompt.root; cd /var/portbuild/${arch}/${branch}
    &prompt.root; rsync -r -v -l -p -c -n distfiles/ portmgr@ftp-master:w/ports/distfiles/ | tee log

ALWAYS use `-n` first with `rsync` and check the output to make sure it is sane. If it looks good, re-run the `rsync` without the `-n` option.

## Experimental Patches Builds

Experimental patches builds are run from time to time to new features or bugfixes to the ports infrastructure (i.e. `bsd.port.mk`), or to test large sweeping upgrades. The current experimental patches branch is `4-exp` on the &i386; architecture.

In general, an experimental patches build is run the same way as any other build. However, before running the `dopackages` script, you must apply the required patches to the ports tree. It is always a good idea to save original copies of all changed files, as well as a list of what you are changing. You can then look back on this list when doing the final commit.

In order to have a good control case with which to compare failures, you should first do a package build of the branch on which the experimental patches branch is based for the &i386; architecture (currently this is `4`). Then, when preparing for the experimental patches build, checkout a ports tree and a src tree with the same date as was used for the control build. This will ensure an apples-to-apples comparison later.

One build cluster can do the control build while the other does the experimental patches build. This can be a great time-saver.

Once the build finishes, compare the control build failures to those of the experimental patches build. Use the following commands to facilitate this (this assumes the `4` branch is the control branch, and the `4-exp` branch is the experimental patches branch):

    &prompt.user; cd /var/portbuild/i386/4-exp/errors
    &prompt.user; find . -name \*.log\* | sort > /tmp/4-exp-errs
    &prompt.user; cd /var/portbuild/i386/4/errors
    &prompt.user; find . -name \*.log\* | sort > /tmp/4-errs

If it has been a long time since one of the builds finished, the logs may have been automatically compressed with bzip2. In that case, you must use `sort | sed 's,\.bz2,,g'` instead.

    &prompt.user; comm -3 /tmp/4-errs /tmp/4-exp-errs | less

This last command will produce a two-column report. The first column is ports that failed on the control build but not in the experimental patches build; the second column is vice versa. Reasons that the port might be in the first column include:

- Port was fixed since the control build was run, or was upgraded to a newer version that is also broken (thus the newer version should appear in the second column)

- Port is fixed by the patches in the experimental patches build

- Port did not build under the experimental patches build due to a dependency failure

Reasons for a port appearing in the second column include:

- Port was broken by the experimental patches [1]

- Port was upgraded since the control build and has become broken [2]

- Port was broken due to a transient error (e.g. FTP site down, package client error, etc.)

Both columns should be investigated and the reason for the errors understood before committing the experimental patches set. To differentiate between [1] and [2] above, you can do a rebuild of the affected packages under the control branch:

    &prompt.user; cd /var/portbuild/i386/4/ports

Be sure to cvs update this tree to the same date as the experimental patches tree.

The following command will set up the control branch for the partial build:

    &prompt.user; /var/portbuild/scripts/dopackages.4 -noportscvs -nobuild -nocvs -nofinish

The builds must be performed from the `packages/All` directory. This directory should initially be empty except for the Makefile symlink. If this symlink does not exist, it must be created:

    &prompt.user; cd /var/portbuild/i386/4/packages/All
    &prompt.user; ln -sf ../../Makefile .
    &prompt.user; make -k -j<#> <list of packages to build>

<#> is the concurrency of the build to attempt. It is usually the sum of the weights listed in `/var/portbuild/i386/mlist` unless you have a reason to run a heavier or lighter build.

The list of packages to build should be a list of package names (including versions) as they appear in `INDEX`. The `PKGSUFFIX` (i.e. .tgz or .tbz) is optional.

This will build only those packages listed as well as all of their dependencies. You can check the progress of this partial build the same way you would a regular build. Once all the errors have been resolved, you can commit the package set. After committing, it is customary to send a `HEADS UP` email to ports@FreeBSD.org and copy ports-developers@FreeBSD.org informing people of the changes. A summary of all changes should also be committed to `/usr/ports/CHANGES`.