---
title: FreeBSD Release Engineering
authors:
  - author: Glen Barber
    email: gjb@FreeBSD.org
organizations:
  - organization: The FreeBSD Foundation
    webpage: https://www.freebsdfoundation.org/
  - organization: Rubicon Communications, LLC (Netgate)
    webpage: https://www.netgate.com/
trademarks: ["freebsd", "intel", "general"]
---

= FreeBSD Release Engineering
:doctype: article
:toc: macro
:toclevels: 1
:icons: font
:sectnums:
:sectnumlevels: 6
:source-highlighter: rouge
:experimental:
:teamBugmeister: FreeBSD Bugmeister Team
:teamDoceng: FreeBSD Documentation Engineering Team
:teamPortmgr: FreeBSD Ports Management Team
:teamPostmaster: FreeBSD Postmaster Team
:teamRe: FreeBSD Release Engineering Team
:teamSecteam: FreeBSD Security Team
:branchHead: head/
:branchStable: stable/
:branchStablex: stable/12/
:branchReleng: releng/
:branchRelengx: releng/12.0/
:branchReleasex: release/12.0.0/
:branchRevision: 12.0

[.abstract-title]
Abstract

This article describes the release engineering process of the FreeBSD Project.

'''

toc::[]

[[introduction]]
== Introduction to the FreeBSD Release Engineering Process

Development of FreeBSD has a very specific workflow. In general, all changes to the FreeBSD base system are committed to the  {branchHead} branch, which reflects the top of the source tree.

After a reasonable testing period, changes can then be merged to the {branchStable} branches. The default minimum timeframe before merging to {branchStable} branches is three (3) days.

Although a general rule to wait a minimum of three days before merging from {branchHead}, there are a few special circumstances where an immediate merge may be necessary, such as a critical security fix, or a bug fix that directly inhibits the release build process.

After several months, and the number of changes in the {branchStable} branch have grown significantly, it is time to release the next version of FreeBSD. These releases have been historically referred to as "point" releases.

In between releases from the {branchStable} branches, approximately every two (2) years, a release will be cut directly from {branchHead}. These releases have been historically referred to as "dot-zero" releases.

This article will highlight the workflow and responsibilities of the {teamRe} for both "dot-zero" and "point"' releases.

The following sections of this article describe:

<<releng-prep>>::
General information and preparation before starting the release cycle.

<<releng-website>>::
Website Changes During the Release Cycle

<<releng-terms>>::
Terminology and general information, such as the "code slush" and "code freeze", used throughout this document.

<<releng-head>>::
The Release Engineering process for a "dot-zero" release.

<<releng-stable>>::
The Release Engineering process for a "point" release.

<<releng-building>>::
Information related to the specific procedures to build installation medium.

<<releng-mirrors>>::
Procedures to publish installation medium.

<<releng-wrapup>>::
Wrapping up the release cycle.

[[releng-prep]]
== General Information and Preparation

Approximately two months before the start of the release cycle, the {teamRe} decides on a schedule for the release. The schedule includes the various milestone points of the release cycle, such as freeze dates, branch dates, and build dates. For example:

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| Milestone
| Anticipated Date

|{branchHead} slush:
|May 27, 2016

|{branchHead} freeze:
|June 10, 2016

|{branchHead} KBI freeze:
|June 24, 2016

|`doc/` tree slush [1]:
|June 24, 2016

|Ports quarterly branch [2]:
|July 1, 2016

|{branchStablex} branch:
|July 8, 2016

|`doc/` tree tag [3]:
|July 8, 2016

|BETA1 build starts:
|July 8, 2016

|{branchHead} thaw:
|July 9, 2016

|BETA2 build starts:
|July 15, 2016

|BETA3 build starts [*]:
|July 22, 2016

|{branchRelengx} branch:
|July 29, 2016

|RC1 build starts:
|July 29, 2016

|{branchStablex} thaw:
|July 30, 2016

|RC2 build starts:
|August 5, 2016

|Final Ports package builds [4]:
|August 6, 2016

|Ports release tag:
|August 12, 2016

|RC3 build starts [*]:
|August 12, 2016

|RELEASE build starts:
|August 19, 2016

|RELEASE announcement:
|September 2, 2016
|===

[NOTE]
====
Items marked with "[*]" are "as needed".
====

. The `doc/` tree slush is coordinated by the {teamDoceng}.
. The Ports quarterly branch used is determined by when the final `RC` build is planned. A new quarterly branch is created on the first day of the quarter, so this metric should be used when taking the release cycle milestones into account. The quarterly branch is created by the {teamPortmgr}.
. The `doc/` tree is tagged by the {teamDoceng}.
. The final Ports package build is done by the {teamPortmgr} after the final (or what is expected to be final) `RC` build.

[NOTE]
====
If the release is being created from an existing {branchStable} branch, the KBI freeze date can be excluded, since the KBI is already considered frozen on established {branchStable} branches.
====

When writing the release cycle schedule, a number of things need to be taken into consideration, in particular milestones where the target date depends on predefined milestones upon which there is a dependency. For example, the Ports Collection release tag originates from the active quarterly branch at the time of the last `RC`. This in part defines which quarterly branch is used, when the release tag can happen, and what revision of the ports tree is used for the final `RELEASE` build.

After general agreement on the schedule, the {teamRe} emails the schedule to the FreeBSD Developers.

It is somewhat typical that many developers will inform the {teamRe} about various works-in-progress. In some cases, an extension for the in-progress work will be requested, and in other cases, a request for "blanket approval" to a particular subset of the tree will be made.

When such requests are made, it is important to make sure timelines (even if estimated) are discussed. For blanket approvals, the length of time for the blanket approval should be made clear. For example, a FreeBSD developer may request blanket approvals from the start of the code slush until the start of the `RC` builds.

[NOTE]
====
In order to keep track of blanket approvals, the {teamRe} uses an internal repository to keep a running log of such requests, which defines the area upon which a blanket approval was granted, the author(s), when the blanket approval expires, and the reason the approval was granted. One example of this is granting blanket approval to [.filename]#release/doc/# to all {teamRe} members until the final `RC` to update the release notes and other release-related documentation.
====

[NOTE]
====
The {teamRe} also uses this repository to track pending approval requests that are received just prior to starting various builds during the release cycle, which the Release Engineer specifies the cutoff period with an email to the FreeBSD developers.
====

Depending on the underlying set of code in question, and the overall impact the set of code has on FreeBSD as a whole, such requests may be approved or denied by the {teamRe}.

The same applies to work-in-progress extensions. For example, in-progress work for a new device driver that is otherwise isolated from the rest of the tree may be granted an extension. A new scheduler, however, may not be feasible, especially if such dramatic changes do not exist in another branch.

The schedule is also added to the Project website, in the `doc/` repository, in [.filename]#head/en_US.ISO8859-1/htdocs/releases/{branchRevision}R/schedule.xml#. This file is continuously updated as the release cycle progresses.

[NOTE]
====
In most cases, the [.filename]#schedule.xml# can be copied from a prior release and updated accordingly.
====

In addition to adding [.filename]#schedule.xml# to the website, [.filename]#head/shared/xml/navibar.ent# and [.filename]#head/shared/xml/release.ent# are also updated to add the link to the schedule to various subpages, as well as enabling the link to the schedule on the Project website index page.

The schedule is also linked from [.filename]#head/en_US.ISO8859-1/htdocs/releng/index.xml#.

Approximately one month prior to the scheduled "code slush", the {teamRe} sends a reminder email to the FreeBSD Developers.

Once the first builds of the release cycle are available, update the `beta.local.where` entity in [.filename]#head/en_US.ISO8859-1/htdocs/releases/{branchRevision}R/schedule.xml#. replacing `IGNORE` with `INCLUDE`.

[NOTE]
====
If two parallel release cycles are happening at once, the `beta2.local.where` entity may be used instead.
====

[[releng-terms]]
== Release Engineering Terminology

This section describes some of the terminology used throughout the rest of this document.

[[releng-terms-code-slush]]
=== The Code Slush

Although the code slush is not a hard freeze on the tree, the {teamRe} requests that bugs in the existing code base take priority over new features.

The code slush does not enforce commit approvals to the branch.

[[releng-terms-code-freeze]]
=== The Code Freeze

The code freeze marks the point in time where all commits to the branch require explicit approval from the {teamRe}.

The FreeBSD Subversion repository contains several hooks to perform sanity checks before any commit is actually committed to the tree. One of these hooks will evaluate if committing to a particular branch requires specific approval.

To enforce commit approvals by the {teamRe}, the Release Engineer updates [.filename]#base/svnadmin/conf/approvers#, and commits the change back to the repository. Once this is done, any change to the branch must include an "Approved by:" line in the commit message.

The "Approved by:" line must match the second column in [.filename]#base/svnadmin/conf/approvers#, otherwise the commit will be rejected by the repository hooks.

[NOTE]
====
During the code freeze, FreeBSD committers are urged to follow the link:https://wiki.freebsd.org/Releng/ChangeRequestGuidelines[Change Request Guidelines].
====

[[releng-terms-kbi-freeze]]
=== The KBI/KPI Freeze

KBI/KPI stability implies that the caller of a function across two different releases of software that implement the function results in the same end state. The caller, whether it is a process, thread, or function, expects the function to operate in a certain way, otherwise the KBI/KPI stability on the branch is broken.

[[releng-website]]
== Website Changes During the Release Cycle

This section describes the changes to the website that should occur as the release cycle progresses.

[NOTE]
====
The files specified throughout this section are relative to the `head/` branch of the `doc` repository in Subversion.
====

[[releng-website-prerelease]]
=== Website Changes Before the Release Cycle Begins

When the release cycle schedule is available, these files need to be updated to enable various different functionalities on the FreeBSD Project website:

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| File to Edit
| What to Change

|[.filename]#share/xml/release.ent#
|Change `beta.upcoming` from `IGNORE` to `INCLUDE`

|[.filename]#share/xml/release.ent#
|Change `% beta.upcoming` from `IGNORE` to `INCLUDE`

|[.filename]#share/xml/release.ent#
|Change `beta.testing` from `IGNORE` to `INCLUDE`

|[.filename]#share/xml/release.ent#
|Change `% beta.testing` from `IGNORE` to `INCLUDE`
|===

[[releng-website-beta-rc]]
=== Website Changes During `BETA` or `RC`

When transitioning from `PRERELEASE` to `BETA`, these files need to be updated to enable the "Help Test" block on the download page. All files are relative to [.filename]#head/# in the `doc` repository:

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| File to Edit
| What to Change

|[.filename]#en_US.ISO8859-1/htdocs/releases/12.0R/schedule.xml#
|Change `% beta.local.where` `IGNORE` to `INCLUDE`

|[.filename]#share/xml/release.ent#
|Update `% betarel.vers` to `BETA__1__`

|[.filename]#share/xml/news.xml#
|Add an entry announcing the `BETA`

|[.filename]#en_US.ISO8859-1/htdocs/security/advisory-template.txt#
|Add the new `BETA`, `RC`, or final `RELEASE` to the template

|[.filename]#en_US.ISO8859-1/htdocs/security/errata-template.txt#
|Add the new `BETA`, `RC`, or final `RELEASE` to the template
|===

Once the {branchRelengx} branch is created, the various release-related documents need to be generated and manually added to the `doc/` repository.

Within [.filename]#release/doc#, invoke to generate [.filename]#errata.html#, [.filename]#hardware.html#, [.filename]#readme.html#, and [.filename]#relnotes.html# pages, which are then added to [.filename]#doc/head/en_US.ISO8859-1/htdocs/releases/X.YR/#, where _X.Y_ represents the major and minor version number of the release.

The `fbsd:nokeywords` property must be set to `on` on the newly-added files before the pre-commit hooks will allow them to be added to the repository.

[NOTE]
====
The relevant release-related documents exist in the [.filename]#doc# repository for FreeBSD 12.x and later.
====

[[releng-ports-beta-rc]]
=== Ports Changes During `BETA`, `RC`, and the Final `RELEASE`

For each build during the release cycle, the `MANIFEST` files containing the `SHA256` of the various distribution sets, such as `base.txz`, `kernel.txz`, and so on, are added to the package:misc/freebsd-release-manifests[] port. This allows utilities other than , such as package:ports-mgmt/poudriere[], to safely use these distribution sets by providing a mechanism through which the checksums can be verified.

[[releng-head]]
== Release from {branchHead}

This section describes the general procedures of the FreeBSD release cycle from the {branchHead} branch.

[[releng-head-builds-alpha]]
=== FreeBSD "`ALPHA`" Builds

Starting with the FreeBSD 10.0-RELEASE cycle, the notion of "`ALPHA`" builds was introduced. Unlike the `BETA` and `RC` builds, `ALPHA` builds are not included in the FreeBSD Release schedule.

The idea behind `ALPHA` builds is to provide regular FreeBSD-provided builds before the creation of the {branchStable} branch.

FreeBSD `ALPHA` snapshots should be built approximately once a week.

For the first `ALPHA` build, the `BRANCH` value in [.filename]#sys/conf/newvers.sh# needs to be changed from `CURRENT` to `ALPHA1`. For subsequent `ALPHA` builds, increment each `ALPHA__N__` value by one.

See <<releng-building>> for information on building the `ALPHA` images.

[[releng-head-branching]]
=== Creating the {branchStablex} Branch

When creating the {branchStable} branch, several changes are required in both the new {branchStable} branch and the {branchHead} branch. The files listed are relative to the repository root. To create the new {branchStablex} branch in Subversion:

[source,bash,subs="attributes"]
....
% svn cp ^/head {branchStablex}
....

Once the {branchStablex} branch has been committed, make the following edits:

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| File to Edit
| What to Change

|[.filename]#stable/12/UPDATING#
|Update the FreeBSD version, and remove the notice about `WITNESS`

|[.filename]#stable/12/contrib/jemalloc/include/jemalloc/jemalloc_FreeBSD.h#
a|

[source,bash,subs="attributes"]
....
#ifndef MALLOC_PRODUCTION
#define MALLOC_PRODUCTION
#endif
....

|[.filename]#stable/12/lib/clang/llvm.build.mk#
|Uncomment `-DNDEBUG`

|[.filename]#stable/12/sys/\*/conf/GENERIC*#
|Remove debugging support

|[.filename]#stable/12/sys/*/conf/MINIMAL#
|Remove debugging support

|[.filename]#stable/12/release/release.conf.sample#
|Update `SRCBRANCH`

|[.filename]#stable/12/sys/*/conf/GENERIC-NODEBUG#
|Remove these kernel configurations

|[.filename]#stable/12/sys/arm/conf/std.arm*#
|Remove debugging options

|[.filename]#stable/12/sys/conf/newvers.sh#
|Update the `BRANCH` value to reflect `BETA1`

|[.filename]#stable/12/shared/mk/src.opts.mk#
|Move `REPRODUCIBLE_BUILD` from `\__DEFAULT_NO_OPTIONS` to `__DEFAULT_YES_OPTIONS`

|[.filename]#stable/12/shared/mk/src.opts.mk#
|Move `LLVM_ASSERTIONS` from `\__DEFAULT_YES_OPTIONS` to `__DEFAULT_NO_OPTIONS` (FreeBSD 13.x and later only)

|[.filename]#stable/12/libexec/rc/rc.conf#
|Set `dumpdev` from `AUTO` to `NO` (it is configurable via for those that want it enabled by default)

|[.filename]#stable/12/release/Makefile#
|Remove the `debug.witness.trace` entries
|===

Then in the {branchHead} branch, which will now become a new major version:

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| File to Edit
| What to Change

|[.filename]#head/UPDATING#
|Update the FreeBSD version

|[.filename]#head/sys/conf/newvers.sh#
|Update the `BRANCH` value to reflect `CURRENT`, and increment `REVISION`

|[.filename]#head/Makefile.inc1#
|Update `TARGET_TRIPLE` and `MACHINE_TRIPLE`

|[.filename]#head/sys/sys/param.h#
|Update `__FreeBSD_version`

|[.filename]#head/gnu/usr.bin/cc/cc_tools/freebsd-native.h#
|Update `FBSD_MAJOR` and `FBSD_CC_VER`

|[.filename]#head/contrib/gcc/config.gcc#
|Append the `freebsdversion.h` section

|[.filename]#head/lib/clang/llvm.build.mk#
|Update the value of `OS_VERSION`

|[.filename]#head/lib/clang/freebsd_cc_version.h#
|Update `FREEBSD_CC_VERSION`

|[.filename]#head/lib/clang/include/lld/Common/Version.inc#
|Update `LLD_REVISION_STRING`

|[.filename]#head/Makefile.libcompat#
|Update `LIB32CPUFLAGS`
|===

[[releng-stable]]
== Release from {branchStable}

This section describes the general procedures of the FreeBSD release cycle from an extablished {branchStable} branch.

[[releng-stable-slush]]
=== FreeBSD `stable` Branch Code Slush

In preparation for the code freeze on a `stable` branch, several files need to be updated to reflect the release cycle is officially in progress. These files are all relative to the top-most level of the stable branch:

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| File to Edit
| What to Change

|[.filename]#sys/conf/newvers.sh#
|Update the `BRANCH` value to reflect `PRERELEASE`

|[.filename]#Makefile.inc1#
|Update `TARGET_TRIPLE`

|[.filename]#lib/clang/llvm.build.mk#
|Update `OS_VERSION`

|[.filename]#Makefile.libcompat#
|Update `LIB32CPUFLAGS`

|[.filename]#gnu/usr.bin/groff/tmac/mdoc.local.in#
|Add a new `.ds` entry for the FreeBSD version, and update `doc-default-operating-system` (FreeBSD 11.x and earlier only)
|===

In the `doc` repository, also update [.filename]#head/en_US.ISO8859-1/htdocs/releases/12.0R/Makefile.hardware#, switching the value of `_BRANCH` to `BETA__X__`, `RC__X__`, or `RELEASE`, respectively.

[[releng-stable-builds-beta]]
=== FreeBSD `BETA` Builds

Following the code slush, the next phase of the release cycle is the code freeze. This is the point at which all commits to the stable branch require explicit approval from the {teamRe}. This is enforced by pre-commit hooks in the Subversion repository by editing [.filename]#base/svnadmin/conf/approvers# to include a regular expression matching the {branchStablex} branch for the release:

[.programlisting,subs="attributes"]
....
^/{branchStablex}	re
^/{branchRelengx}	re
....

[NOTE]
====
There are two general exceptions to requiring commit approval during the release cycle. The first is any change that needs to be committed by the Release Engineer in order to proceed with the day-to-day workflow of the release cycle, the other is security fixes that may occur during the release cycle.
====

Once the code freeze is in effect, the next build from the branch is labeled `BETA1`. This is done by updating the `BRANCH` value in [.filename]#sys/conf/newvers.sh# from `PRERELEASE` to `BETA1`.

Once this is done, the first set of `BETA` builds are started. Subsequent `BETA` builds do not require updates to any files other than [.filename]#sys/conf/newvers.sh#, incrementing the `BETA` build number.

[[releng-stable-branching]]
=== Creating the {branchRelengx} Branch

When the first `RC` (Release Candidate) build is ready to begin, the {branchReleng} branch is created. This is a multi-step process that must be done in a specific order, in order to avoid anomalies such as overlaps with `__FreeBSD_version` values, for example. The paths listed below are relative to the repository root. The order of commits and what to change are:

[source,bash,subs="attributes"]
....
% svn cp ^/{branchStablex} {branchRelengx}
....

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| File to Edit
| What to Change

|[.filename]#releng/12.0/sys/conf/newvers.sh#
|Change `BETA__X__` to `RC1`

|[.filename]#releng/12.0/sys/sys/param.h#
|Update `__FreeBSD_version`

|[.filename]#releng/12.0/etc/pkg/FreeBSD.conf#
|Replace `latest` with `quarterly` as the default package repository location

|[.filename]#releng/12.0/release/pkg_repos/release-dvd.conf#
|Replace `latest` with `quarterly` as the default package repository location

|[.filename]#stable/12/sys/conf/newvers.sh#
|Update `BETA__X__` with `PRERELEASE`

|[.filename]#stable/12/sys/sys/param.h#
|Update `__FreeBSD_version`

|[.filename]#svnadmin/conf/approvers#
|Add a new approvers line for the releng branch as was done for the stable branch
|===

[source,bash,subs="attributes"]
....
% svn propdel -R svn:mergeinfo {branchRelengx}
% svn commit {branchRelengx}
% svn commit {branchStablex}
....

Now that two new `__FreeBSD_version` values exist, also update [.filename]#head/en_US.ISO8859-1/books/porters-handbook/versions/chapter.xml# in the Documentation Project repository.

After the first `RC` build has completed and tested, the {branchStable} branch can be "thawed" by removing (or commenting) the ^/{branchStablex} entry in [.filename]#svnadmin/conf/approvers#.

Following the availability of the first `RC`, {teamBugmeister} should be emailed to add the new FreeBSD `-RELEASE` to the `versions` available in the drop-down menu shown in the bug tracker.

[[releng-building]]
== Building FreeBSD Installation Media

This section describes the general procedures producing FreeBSD development snapshots and releases.

[[releng-build-scripts]]
=== Release Build Scripts

This section describes the build scripts used by {teamRe} to produce development snapshots and releases.

[[releng-build-scripts-single]]
==== The [.filename]#release.sh# Script

Prior to FreeBSD 9.0-RELEASE, [.filename]#src/release/Makefile# was updated to support , and the [.filename]#src/release/generate-release.sh# script was introduced as a wrapper to automate invoking the targets.

Prior to FreeBSD 9.2-RELEASE, [.filename]#src/release/release.sh# was introduced, which heavily based on [.filename]#src/release/generate-release.sh# included support to specify configuration files to override various options and environment variables. Support for configuration files provided support for cross building each architecture for a release by specifying a separate configuration file for each invocation.

As a brief example of using [.filename]#src/release/release.sh# to build a single release in [.filename]#/scratch#:

[source,bash,subs="attributes"]
....
# /bin/sh /usr/src/release/release.sh
....

As a brief example of using [.filename]#src/release/release.sh# to build a single, cross-built release using a different target directory, create a custom [.filename]#release.conf# containing:

[.programlisting,subs="attributes"]
....
# release.sh configuration for powerpc/powerpc64
CHROOTDIR="/scratch-powerpc64"
TARGET="powerpc"
TARGET_ARCH="powerpc64"
KERNEL="GENERIC64"
....

Then invoke [.filename]#src/release/release.sh# as:

[source,bash,subs="attributes"]
....
# /bin/sh /usr/src/release/release.sh -c $HOME/release.conf
....

See and [.filename]#src/release/release.conf.sample# for more details and example usage.

[[releng-build-scripts-multiple]]
==== The [.filename]#thermite.sh# Wrapper Script

In order to make cross building the full set of architectures supported on a given branch faster, easier, and reduce human error factors, a wrapper script around [.filename]#src/release/release.sh# was written to iterate through the various combinations of architectures and invoke [.filename]#src/release/release.sh# using a configuration file specific to that architecture.

The wrapper script is called [.filename]#thermite.sh#, which is available in the FreeBSD Subversion repository at `svn://svn.freebsd.org/base/user/gjb/thermite/`, in addition to configuration files used to build {branchHead} and {branchStablex} development snapshots.

Using [.filename]#thermite.sh# is covered in <<releng-build-snapshot>> and <<releng-build-release>>.

Each architecture and individual kernel have their own configuration file used by [.filename]#release.sh#. Each branch has its own [.filename]#defaults-X.conf# configuration which contains entries common throughout each architecture, where overrides or special variables are set and/or overridden in the per-build files.

The per-build configuration file naming scheme is in the form of [.filename]#${revision}-${TARGET_ARCH}-${KERNCONF}-${type}.conf#, where the uppercase variables are equivalent to what uses in the build system, and lowercase variables are set within the configuration files, mapping to the major version of the respective branch.

Each branch also has its own [.filename]#builds-X.conf# configuration, which is used by [.filename]#thermite.sh#. The [.filename]#thermite.sh# script iterates through each ${revision}, ${TARGET_ARCH}, ${KERNCONF}, and ${type} value, creating a master list of what to build. However, a given combination from the list will only be built if the respective configuration file exists, which is where the naming scheme above is relevant.

There are two paths of file sourcing:

* [.filename]#builds-12.conf# - [.filename]#main.conf#
+ 
This controls [.filename]#thermite.sh# behavior
* [.filename]#12-amd64-GENERIC-snap.conf# - [.filename]#defaults-12.conf# - [.filename]#main.conf#
+ 
This controls [.filename]#release/release.sh# behavior within the build 

[NOTE]
====
The [.filename]#builds-12.conf#, [.filename]#defaults-12.conf#, and [.filename]#main.conf# configuration files exist to reduce repetition between the various per-build files.
====

[[releng-build-snapshot]]
=== Building FreeBSD Development Snapshots

The official release build machines have a specific filesystem layout, which using ZFS, [.filename]#thermite.sh# takes heavy advantage of with clones and snapshots, ensuring a pristine build environment.

The build scripts reside in [.filename]#/releng/scripts-snapshot/scripts# or [.filename]#/releng/scripts-release/scripts# respectively, to avoid collisions between an `RC` build from a releng branch versus a `STABLE` snapshot from the respective stable branch.

A separate dataset exists for the final build images, [.filename]#/snap/ftp#. This directory contains both snapshots and releases directories. They are only used if the `EVERYTHINGISFINE` variable is defined in [.filename]#main.conf#.

[NOTE]
====
The `EVERYTHINGISFINE` variable name was chosen to avoid colliding with a variable that might be possibly set in the user environment, accidentally enabling the behavior that depends on it being defined.
====

As [.filename]#thermite.sh# iterates through the master list of combinations and locates the per-build configuration file, a ZFS dataset is created under [.filename]#/releng#, such as [.filename]#/releng/12-amd64-GENERIC-snap#. The `src/`, `ports/`, and `doc/` trees are checked out to separate ZFS datasets, such as [.filename]#/releng/12-src-snap#, which are then cloned and mounted into the respective build datasets. This is done to avoid checking out a given tree more than once.

Assuming these filesystem paths, [.filename]#thermite.sh# would be invoked as:

[source,bash,subs="attributes"]
....
# cd /releng/scripts-snapshot/scripts
# ./setrev.sh -b {branchStablex}
# ./zfs-cleanup.sh -c ./builds-12.conf
# ./thermite.sh -c ./builds-12.conf
....

Once the builds have completed, additional helper scripts are available to generate development snapshot emails which are sent to the `freebsd-snapshots@freebsd.org` mailing list:

[source,bash,subs="attributes"]
....
# cd /releng/scripts-snapshot/scripts
# ./get-checksums.sh -c ./builds-12.conf | ./generate-email.pl > snapshot-12-mail
....

[NOTE]
====
The generated output should be double-checked for correctness, and the email itself should be PGP signed, in-line.
====

[NOTE]
====
These helper scripts only apply to development snapshot builds. Announcements during the release cycle (excluding the final release announcement) are created from an email template. A sample of the email template currently used can be found link:here[here].
====

[[releng-build-release]]
=== Building FreeBSD Releases

Similar to building FreeBSD development snapshots, [.filename]#thermite.sh# would be invoked the same way. The difference between development snapshots and release builds, `BETA` and `RC` included, is that the configuration files must be named with `release` instead of `snap` as the type, as mentioned above.

In addition, the `BUILDTYPE` and `types` must be changed from `snap` to `release` in [.filename]#defaults-12.conf# and [.filename]#builds-12.conf#, respectively.

When building `BETA`, `RC`, and the final `RELEASE`, also statically set `BUILDSVNREV` to the revision on the branch reflecting the name change, `BUILDDATE` to the date the builds are started in `YYYYMMDD` format. If the `doc/` and `ports/` trees have been tagged, also set `PORTBRANCH` and `DOCBRANCH` to the relevant tag path in the Subversion repository, replacing `HEAD` with the last changed revision. Also set `releasesrc` in [.filename]#builds-12.conf# to the relevant branch, such as {branchStablex} or {branchRelengx}.

During the release cycle, a copy of [.filename]#CHECKSUM.SHA512# and [.filename]#CHECKSUM.SHA256# for each architecture are stored in the {teamRe} internal repository in addition to being included in the various announcement emails. Each [.filename]#MANIFEST# containing the hashes of [.filename]#base.txz#, [.filename]#kernel.txz#, etc. are added to package:misc/freebsd-release-manifests[] in the Ports Collection, as well.

In preparation for the release build, several files need to be updated:

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| File to Edit
| What to Change

|[.filename]#sys/conf/newvers.sh#
|Update the `BRANCH` value to `RELEASE`

|[.filename]#UPDATING#
|Add the anticipated announcement date

|[.filename]#lib/csu/common/crtbrand.c#
|Replace `__FreeBSD_version` with the value in [.filename]#sys/sys/param.h#
|===

After building the final `RELEASE`, the {branchRelengx} branch is tagged as {branchReleasex} using the revision from which the `RELEASE` was built. Similar to creating the {branchStablex} and {branchRelengx} branches, this is done with `svn cp`. From the repository root:

[source,bash,subs="attributes"]
....
% svn cp ^/{branchRelengx}@r306420 {branchReleasex}
% svn commit {branchReleasex}
....

[[releng-mirrors]]
== Publishing FreeBSD Installation Media to Project Mirrors

This section describes the procedure to publish FreeBSD development snapshots and releases to the Project mirrors.

[[releng-mirrors-staging]]
=== Staging FreeBSD Installation Media Images

Staging FreeBSD snapshots and releases is a two part process:

* Creating the directory structure to match the hierarchy on `ftp-master`
+ 
If `EVERYTHINGISFINE` is defined in the build configuration files, [.filename]#main.conf# in the case of the build scripts referenced above, this happens automatically in the after the build is complete, creating the directory structure in [.filename]#${DESTDIR}/R/ftp-stage# with a path structure matching what is expected on `ftp-master`. This is equivalent to running the following in the directly:
+
[source,bash,subs="attributes"]
....
# make -C /usr/src/release -f Makefile.mirrors EVERYTHINGISFINE=1 ftp-stage
....
+ 
After each architecture is built, [.filename]#thermite.sh# will rsync the [.filename]#${DESTDIR}/R/ftp-stage# from the build to [.filename]#/snap/ftp/snapshots# or [.filename]#/snap/ftp/releases# on the build host, respectively.
* Copying the files to a staging directory on `ftp-master` before moving the files into [.filename]#pub/# to begin propagation to the Project mirrors
+ 
Once all builds have finished, [.filename]#/snap/ftp/snapshots#, or [.filename]#/snap/ftp/releases# for a release, is polled by `ftp-master` using rsync to [.filename]#/archive/tmp/snapshots# or [.filename]#/archive/tmp/releases#, respectively.
+
[NOTE]
====
On `ftp-master` in the FreeBSD Project infrastructure, this step requires `root` level access, as this step must be executed as the `archive` user.
====

[[releng-mirrors-publishing]]
=== Publishing FreeBSD Installation Media

Once the images are staged in [.filename]#/archive/tmp/#, they are ready to be made public by putting them in [.filename]#/archive/pub/FreeBSD#. In order to reduce propagation time, is used to create hard links from [.filename]#/archive/tmp# to [.filename]#/archive/pub/FreeBSD#.

[NOTE]
====
In order for this to be effective, both [.filename]#/archive/tmp# and [.filename]#/archive/pub# must reside on the same logical filesystem.
====

There is a caveat, however, where rsync must be used after in order to correct the symbolic links in [.filename]#pub/FreeBSD/snapshots/ISO-IMAGES# which will replace with a hard link, increasing the propagation time.

[NOTE]
====
As with the staging steps, this requires `root` level access, as this step must be executed as the `archive` user.
====

As the `archive` user:

[source,bash,subs="attributes"]
....
% cd /archive/tmp/snapshots
% pax -r -w -l . /archive/pub/FreeBSD/snapshots
% /usr/local/bin/rsync -avH /archive/tmp/snapshots/* /archive/pub/FreeBSD/snapshots/
....

Replace _snapshots_ with _releases_ as appropriate.

[[releng-wrapup]]
== Wrapping up the Release Cycle

This section describes general post-release tasks.

[[releng-wrapup-en]]
=== Post-Release Errata Notices

As the release cycle approaches conclusion, it is common to have several EN (Errata Notice) candidates to address issues that were discovered late in the cycle. Following the release, the {teamRe} and the {teamSecteam} revisit changes that were not approved prior to the final release, and depending on the scope of the change in question, may issue an EN.

[NOTE]
====
The actual process of issuing ENs is handled by the {teamSecteam}.
====

To request an Errata Notice after a release cycle has completed, a developer should fill out the https://www.freebsd.org/security/errata-template.txt[Errata Notice template], in particular the `Background`, `Problem Description`, `Impact`, and if applicable, `Workaround` sections.

The completed Errata Notice template should be emailed together with either a patch against the {branchReleng} branch or a list of revisions from the {branchStable} branch.

For Errata Notice requests immediately following the release, the request should be emailed to both the {teamRe} and the {teamSecteam}. Once the {branchReleng} branch has been handed over to the {teamSecteam} as described in <<releng-wrapup-handoff>>, Errata Notice requests should be sent to the {teamSecteam}.

[[releng-wrapup-handoff]]
=== Handoff to the {teamSecteam}

Roughly two weeks following the release, the Release Engineer updates [.filename]#svnadmin/conf/approvers# changing the approver column from `re` to `(so|security-officer)` for the {branchRelengx} branch.

[[releng-eol]]
== Release End-of-Life

This section describes the website-related files to update when a release reaches EoL (End-of-Life).

[[releng-eol-website]]
=== Website Updates for End-of-Life

When a release reaches End-of-Life, references to that release should be removed and/or updated on the website:

[.informaltable]
[cols="1,1", frame="none", options="header"]
|===
| File
| What to Change


|[.filename]#head/en_US.ISO8859-1/htdocs/index.xsl#
|Remove `&u.relXXX.announce;` and `&u.relXXX.current;` references.

|[.filename]#head/en_US.ISO8859-1/htdocs/releases/index.xml#
|Move the &u.relXXX.*; macros from the supported release list to the Legacy Releases list.

|[.filename]#head/en_US.ISO8859-1/htdocs/releng/index.xml#
|Update the appropriate releng branch to refelect the branch is no longer supported.

|[.filename]#head/en_US.ISO8859-1/htdocs/security/security.xml#
|Remove the branch from the supported branch list.

|[.filename]#head/en_US.ISO8859-1/htdocs/where.xml#
|Remove the URLs for the release.

|[.filename]#head/share/xml/navibar.ent#
|Remove `&u.relXXX.announce;` and `&u.relXXX.current;` references.

|[.filename]#head/en_US.ISO8859-1/htdocs/security/advisory-template.txt#
|Remove references to the release and releng branch.

|[.filename]#head/en_US.ISO8859-1/htdocs/security/errata-template.txt#
|Remove references to the release and releng branch.
|===
