# FreeBSD From Scratch

Jens
Schweikhardt

    schweikh@FreeBSD.org

2002,2003,2004
Jens Schweikhardt
$FreeBSD$
&tm-attrib.freebsd; &tm-attrib.adobe; &tm-attrib.general;

This article describes my efforts at &scratch.ap;: a fully automated installation of a customized &os; system compiled from source, including compilation of all your favorite ports and configured to match your idea of the perfect system. If you think `make world` is a wonderful concept, &scratch.ap; extends it to `make evenmore`.

## Introduction

Have you ever upgraded your system with `make world`? There is a problem if you have only one system on your disks. If the installworld fails partway through, you are left with a broken system that might not even boot any longer. Or maybe the installworld runs smoothly but the new kernel does not boot. Then it is time to reach for the Fixit CD and dig for those backups you have taken half a year ago.

I believe in the wipe your disks when upgrading systems paradigm. Wiping disks, or rather partitions, makes sure there is no old cruft left lying around, something which most upgrade procedures just do not care about. But wiping the partitions means you have to also recompile/reinstall all your ports and packages and then redo all your carefully crafted configuration tweaks. If you think that this task should be automated as well, read on.

## Why would I (not) want &scratch.ap;?

This is a legitimate question. We have sysinstall and the well known way to compile the kernel and the userland tools.

The problem with sysinstall is that it is severely limited in what, where and how it can install.

- It is normally used to install pre-built distribution sets and packages from some other source (CD, DVD, FTP). It cannot install the result of a `make buildworld`.

- It cannot install a second system under a directory in a running system.

- It cannot install in Vinum partitions.

- It cannot compile ports, only install precompiled packages.

- It is hard to script or to make arbitrary post-installation changes.

- Last but not least, sysinstall is semi-officially at its End-Of-Life.

The well known way to build and install the world, as described in the Handbook, by default replaces the existing system. Only the kernel and modules are saved. System binaries, headers and a lot of other files are overwritten; obsolete files are still present and can cause surprises. If the upgrade fails for any reason, it may be hard or even impossible to restore the previous state of the system.

&scratch.ap; solves all these problems. The strategy is simple: use a running system to install a new system under an empty directory tree, while new partitions are mounted appropriately in that tree. Many config files can be copied to the appropriate place and &man.mergemaster.8; can take care of those that cannot. Arbitrary post-configuration of the new system can be done from within the old system, up to the point where you can chroot to the new system. In other words, we go through three stages, where each stage consists of either running a shell script or invoke `make`:

1. `stage_1.sh`: Create a new bootable system under an empty directory and merge or copy as many files as are necessary. Then boot the new system.

2. `stage_2.sh`: Install desired ports.

3. `stage_3.mk`: Do post-configuration for software installed in previous stage.

Once you have used &scratch.ap; to build a second system and found it works satisfactorily for a couple of weeks, you can then use it again to reinstall the original system. From now on, whenever you feel like an update is in order, you simply toggle the partitions you want to wipe and reinstall.

Maybe you have heard of or even tried Linux From Scratch, or LFS for short. LFS also describes how to build and install a system from scratch in empty partitions using a running system. The focus in LFS seems to be to show the role of each system component (such as kernel, compiler, devices, shell, terminal database, etc) and the details of each component's installation. &scratch.ap; does not go into that much detail. My goal is to provide an automated and complete installation, not explaining all the gory details that go on under the hood when making the world. In case you want to explore &os; at this level of detail, start looking at `/usr/src/Makefile` and follow the actions of a `make buildworld`.

There are also downsides in the approach taken by &scratch.ap; that you should bear in mind.

- While compiling the ports during stage two the system can not be used for its usual duties. If you run a production server you have to consider the downtime caused by stage two. The ports compiled by `stage_2.conf.default` below require about 4 hours to build on an AMD1800+ SCSI system with 10krpm disks and 1GB of RAM. If you prefer to install packages instead of ports, you can significantly reduce the downtime to about 10 minutes.

## Prerequisites

For going the &scratch.ap; way, you need to have:

- A running &os; system with sources and a ports tree.

- At least one unused partition where the new system will be installed.

- Experience with running &man.mergemaster.8;. Or at least no fear doing so.

- If you have no or only a slow link to the Internet: the distfiles for your favorite ports.

- Basic knowledge of shell scripting with the Bourne shell, &man.sh.1;.

- Finally, you should also be able to tell your boot loader how to boot the new system, either interactively, or by means of a config file.

## Stage One: System Installation

The first version of this article used a single shell script for stage one where all your customization had to be done by editing the script. After valuable user feedback I have decided to separate the code and data in the scripts. This allows to have different configuration data sets to install different systems without changing any of the code scripts.

The code script for stage one is `stage_1.sh` and when run with exactly one argument, like

    &prompt.root; ./stage_1.sh default

will read its configuration from `stage_1.conf.default` and write a log to `stage_1.log.default`.

Further below you find my `stage_1.conf.default`. You need to customize it in various places to match your idea of the perfect system. I have tried to extensively comment the places you should adapt. The configuration script must provide four shell functions, `create_file_systems`, `create_etc_fstab`, `copy_files` and `all_remaining_customization` (in case it matters: this is also the sequence in which they will be called from `stage_1.sh`).

The points to ponder are:

- Partition layout.

  I do not subscribe to the idea of a single huge partition for the whole system. My systems generally have at least one partition for `/`, `/usr` and `/var` with `/tmp` symlinked to `/var/tmp`. In addition I share the file systems for `/home` (user homes), `/home/ncvs` (&os; CVS repository replica), `/usr/ports` (the ports tree), `/src` (various checked out src trees) and `/share` (other shared data without the need for backups, like the news spool).

- Luxury items.

  What you want immediately after booting the new system and even before starting stage two. The reason for not simply chrooting to the new system during stage one and installing all my beloved ports is that in theory and in practice there are bootstrap and consistency issues: stage one has your old kernel running, but the chrooted environment consists of new binaries and headers. If the new binaries use a new system call, these binaries will die with `SIGSYS, Bad system call`, because the old kernel does not have that system call. I have seen other issues when I tried building `lang/perl5`.

Before you run `stage_1.sh` make sure you have completed the usual tasks in preparation for `make installworld installkernel`, like:

- configured your kernel config file

- successfully completed `make buildworld`

- successfully completed `make buildkernel KERNCONF=whatever`

When you run `stage_1.sh` for the first time, and the config files copied from your running system to the new system are not up-to-date with respect to what is under `/usr/src`, `mergemaster` will ask you how to proceed. I recommend merging the changes. If you get tired of going through the dialogues you can simply update the files on your running system once (Only if this is an option. You probably do not want to do this if one of your systems runs `-STABLE` and the other `-CURRENT`. The changes may be incompatible). Subsequent `mergemaster` invocations will detect that the RCS version IDs match those under `/usr/src` and skip the file.

The `stage_1.sh` script will stop at the first command that fails (returns a non-zero exit status) due to `set -e`, so you cannot overlook errors. It will also stop if you use an unset environment variable, probably due to a typo. You should correct any errors in your version of `stage_1.conf.default` before you go on.

In `stage_1.sh` we invoke `mergemaster`. Even if none of the files requires a merge, it will display and ask at the end

    *** Comparison complete

    Do you wish to delete what is left of /var/tmp/temproot.stage1? [no] no

Please answer `no` or just hit Enter. The reason is that `mergemaster` will have left a few zero sized files below `/var/tmp/temproot.stage1` which will be copied to the new system later (unless already there).

After that it will list the files it installed, making use of a pager, &man.more.1; by default, optionally &man.less.1;:

    *** You chose the automatic install option for files that did not
        exist on your system.  The following were installed for you:
          /newroot/etc/defaults/rc.conf
          ...
          /newroot/COPYRIGHT

    (END)

Type q to quit the pager. Then you will be informed about `login.conf`:

    *** You installed a login.conf file, so make sure that you run
        '/usr/bin/cap_mkdb /newroot/etc/login.conf'
        to rebuild your login.conf database

        Would you like to run it now? y or n [n]

The answer does not matter since we will run &man.cap.mkdb.1; in any case.

Here is the author's `stage_1.conf.default`, which you need to modify substantially. The comments give you enough information what to change.

Please pay attention to the &man.newfs.8; commands. While you can not create new file systems on mounted partitions, the script will happily erase any unmounted `/dev/da0s1a`, `/dev/da0s1e` and `/dev/da2s1e`. This can be enough to ruin your day, so be sure to modify the device names.

Download `stage_1.conf.default` .

Running this script installs a system that when booted provides:

- Inherited users and groups.

- Firewalled Internet connectivity over Ethernet and PPP.

- Correct time zone and NTP.

- Some more minor configuration, like `/etc/ttys` and `inetd`.

Other areas are prepared for configuration, but will not work until stage two is completed. For example we have copied files to configure printing and X11. Printing however is likely to need applications not found in the base system, like &postscript; utilities. X11 will not run before we have compiled the server, libraries and programs.

## Stage Two: Ports Installation

It is also possible to install the (precompiled) packages at this stage, instead of compiling ports. In this case, `stage_2.sh` would be nothing more than a list of `pkg_add` commands. I trust you know how to write such a script. Here we concentrate on the more flexible and traditional way of using the ports.

The following `stage_2.sh` script is how I install my favorite ports. It can be run any number of times and will skip all ports that are already installed. It supports the dryrun option (`-n`) to just show what would be done. You run it like `stage_1.sh` with exactly one argument to denote a config file, e.g.

    &prompt.root; ./stage_2.sh default

which will read the list of ports from `stage_2.conf.default`.

The list of ports consists of lines with two or more space separated words: the category and the port, optionally followed by an installation command that will compile and install the port (default: `make install BATCH=yes < /dev/null`). Empty lines and lines starting with # are ignored. Most of the time it suffices to only name category and port. A few ports however can be fine tuned by specifying `make` variables, e.g.:

    www mozilla make WITHOUT_MAILNEWS=yes WITHOUT_CHATZILLA=yes install

In fact you can specify arbitrary shell commands, so you are not restricted to simple `make` invocations:

    java linux-sun-jdk13 yes | make install
    news inn-stable CONFIGURE_ARGS="--enable-uucp-rnews --enable-setgid-inews" make install

Note that the line for `news/inn-stable` is an example for a one-shot shell variable assignment to `CONFIGURE_ARGS`. The port `Makefile` will use this as an initial value and augment some other essential args. The difference to specifying a make variable on the command line with

    news inn-stable make CONFIGURE_ARGS="--enable-uucp-rnews --enable-setgid-inews" install

is that the latter will override instead of augment. It depends on the particular port which method you want.

Be careful that your ports do not use an interactive install, i.e. they should not try to read from stdin other than what you explicitly give them on stdin. If they do, they will read the next line(s) from your list of ports in the here-document and get confused. If `stage_2.sh` mysteriously skips a port or stops processing, this is likely the reason.

Below is `stage_2.conf.default`. A log file named `LOGDIR/category+port` is created for each port it actually installs.

Download `stage_2.conf.default`.

## Stage Three

You have installed your beloved ports during stage two. Some ports require a little bit of configuration. This is what stage three, the post-configuration is for. I could have integrated this post-configuration at the end of the `stage_2.sh` script. However, I think there is a conceptual difference between installing a port and modifying its out-of-the-box configuration that warrants a separate stage.

I have chosen to implement stage three as a `Makefile` because this allows easy selection of what you want to configure simply by running:

    &prompt.root; make -f stage_3.mk target

As with `stage_2.sh` make sure you have `stage_3.mk` available after booting the new system, either by putting it on a shared partition or copying it somewhere on the new system.

## Limitations

The automated installation of a port may prove difficult if it is interactive and does not support `make BATCH=YES install`. For a few ports the interaction is nothing more than typing `yes` when asked to accept some license. If such input is read from the standard input, we simply pipe the appropriate answers to the installation command (usually `make install`; this is how I deal with `java/linux-sun-jdk14` in `stage_2.conf.default`).

This strategy for example does not work for `editors/staroffice52`, which requires that X11 is running. The installation procedure involves a fair amount of clicking and typing, so it cannot be automated like other ports can. However the following workaround does the trick for me: first I create a staroffice package on the old system with

    &prompt.root; cd /usr/ports/editors/staroffice52
    &prompt.root; make package
    ===>  Building package for staroffice-5.2_1
    Creating package /usr/ports/editors/staroffice52/staroffice-5.2_1.tbz
    Registering depends:.
    Creating bzip'd tar ball in '/usr/ports/editors/staroffice52/staroffice-5.2_1.tbz'

and during stage two I simply use:

    &prompt.root; pkg_add /usr/ports/editors/staroffice52/staroffice-5.2_1.tbz

You should also be aware of upgrade issues for config files. In general you do not know when and if the format or contents of a config file changes. A new group may be added to `/etc/group`, or `/etc/passwd` may gain another field. All of this has happened in the past. Simply copying a config file from the old to the new system may be enough most of the time, but in these cases it was not. If you update a system the canonical way (by overwriting the old files) you are expected to use `mergemaster` to deal with changes where you effectively want to merge your local config with potentially new items. Unfortunately, `mergemaster` is only available for base system files, not for anything installed by ports. Some third party software seems to be especially designed to keep me on my toes by changing the config file format every fortnight. To detect such silent changes, I keep a copy of the modified config files in the same place where I keep `stage_3.mk` and compare the result with a make rule, e.g. for apache's `httpd.conf` in target `config_apache` with

    @if ! cmp -s /usr/local/etc/apache2/httpd.conf httpd.conf; then \
        echo "ATTENTION: the httpd.conf has changed. Please examine if"; \
        echo "the modifications are still correct. Here is the diff:"; \
        diff -u /usr/local/etc/apache2/httpd.conf httpd.conf; \
    fi

If the diff is innocuous I can make the message go away with `cp /usr/local/etc/apache2/httpd.conf httpd.conf`.

I have used &scratch.ap; several times to update a `5-CURRENT` to `5-CURRENT`, i.e. I have never tried to install a `5-CURRENT` from a `4-STABLE` system or vice versa. Due to the number of changes between different major release numbers I would expect this process to be a bit more involved. Using &scratch.ap; for upgrades within the realm of `4-STABLE` should work painlessly (although I have not yet tried it.) Users of `4-STABLE` may want to consider the following areas:

If you do not use the device file system, &man.devfs.5;, you may want to create devices for some of your hardware with &man.MAKEDEV.8; in `all_remaining_customization`.

## The Files

Here are the three files you need beside the config files already shown above.

This is the `stage_1.sh` script, which you should not need to modify.

Download `stage_1.sh`.

This is the `stage_2.sh` script. You may want to modify the variables at the beginning.

Download `stage_2.sh`.

This is my `stage_3.mk` to give you an idea how to automate all reconfiguration.

Download `stage_3.mk`.